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Executive  Summary 

The  Community  STEM  Outreach  Program:  A  Local  Model  for  National  Impact  at  the  Saint  Louis 
Science  Center  (SLSC)  focused  on  three  key  elements  of  expansion:  programmatic  expansion, 
stakeholder  expansion,  and  national  expansion.  The  program  of  focus  was  the  Youth  Exploring  Science 
(YES)  program,  an  out-of-school  time  urban  youth  development  program  initiated  in  1997  to  engage 
underrepresented  teens  in  STEM  inquiry.  For  the  grant  project's  expansion  goals,  YES  educators  and 
leaders  identified  new  programmatic  strengths  and  weaknesses,  maintaining  elements  of  programming 
that  proved  successful  and  reviewing  why  some  promising  changes  did  not  meet  expectations.  Also 
during  this  time,  external  evaluators  evaluated  key  metrics  in  the  Youth  Exploring  Science  (YES)  program 
at  SLSC,  identified  critical  elements  contributing  to  the  program's  impact,  formulated  a  logic  model  to 
encompass  those  elements,  and  created  a  tool  to  guide  further  program  development  at  other  sites 
throughout  the  country. 

Of  the  project's  fourteen  initial  objectives,  seven  have  been  implemented  and  will  continue  to 
impact  program  operations  going  forward.  These  seven  objectives  involved  expanding  the  YES  staff  and 
strengthening  the  role  of  STEM  content  in  program  activities  in  order  to  support  participant 
achievement.  The  seven  ongoing  objectives  also  incorporated  outreach  activities  that  expanded  and 
diversified  the  pool  of  YES  participants.  Outreach  efforts  will  continue  to  engage  counselors  and 
teachers  from  a  diverse  range  of  schools  and  community  organizations.  Staff  development  and 
reflections  will  also  continue,  as  will  "train-the-trainer"  activities  to  expand  the  program's  reach  into 
other  organizations  and  communities. 

Three  additional  objectives  were  completed,  generating  evaluation  and  dissemination  deliverables. 
External  evaluator  Klein  Consulting  produced  an  electronic  and  print  documentation  of  the  YES  program 
for  use  in  national  dissemination  efforts.  This  tool  guides  users  through  a  program  development  process 
rather  than  assume  broad  applicability  of  the  YES  model  as  it  exists  in  St.  Louis.  Klein  also  provided 
guidance  for  continued  research,  and  both  Klein  and  SLSC  leadership  have  expressed  interest  in 
continued  research.  Internal  and  external  evaluation  both  produced  valuable  results,  and  continuing 
internal  evaluation  will  incorporate  elements  previously  examined  by  the  external  evaluator.  Four  of  the 
initial  objectives  presented  challenges  during  implementation  and  were  revised  based  on  new 
information. 

Four  objectives  were  attempted  and  revised  during  implementation.  The  YES  program  welcomed  50 
additional  teens  into  its  program  in  2012,  doubling  the  size  of  its  standard  cohort.  Increased  classroom 
size  presented  challenges  to  maintaining  the  level  of  individualized  attention  so  central  to  YES 
programming.  This  created  inefficiencies  in  participant  retention  and  impact.  Even  though  the  program 
interacted  with  more  students,  these  students  did  not  benefit  as  greatly  from  their  involvement  in  YES. 
In  2013,  the  program  welcomed  a  standard  cohort,  and  programming  will  continue  to  use  this  model. 
Routine  engagement  opportunities  for  Navy  personnel  also  proved  difficult.  The  program  instead 
advocates  involving  Navy  professionals,  as  well  as  other  STEM  professionals  with  significant 
responsibilities,  in  shorter-term  projects  rather  than  ongoing  volunteerism.  The  project's  last  two 
objectives  involving  national  expansion  and  the  resources  needed  for  that  were  also  revised  during  a 
collaborative  conference  in  April  2013.  Several  informal  STEM  leaders  gathered  and,  while  a  basic 
structure  for  YES-based  programming  was  identified,  the  group  found  it  difficult  to  identify  a  model  that 
could  be  broadly  applied.  As  a  result,  Klein  Consulting  focused  primarily  on  the  program  development 
process  in  dissemination  deliverables,  guiding  industry  professionals  to  create  a  STEM  youth  program 
matching  their  local  needs  and  capabilities. 
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Program  Final  Report 

Community  STEM  Outreach  Program:  A  Local  Model  for  National  Impact 
Award  N00014-10-1-1000 

Overview 

Office  of  Naval  Research  funding  for  the  Community  STEM  Outreach  Program  provided  an 
opportunity  to  investigate  several  elements  of  the  Youth  Exploring  Science  (YES)  program  at  the  Saint 
Louis  Science  Center  (SLSC).  The  project  managed  to  identify  program  strengths,  weather  significant 
programmatic  change  and  assemble  a  dissemination  kit  to  create  or  enhance  similar  youth  development 
STEM  programs  nationally. 

Project  Goals 

The  Community  STEM  Outreach  Program  proposal  identified  three  central  components  to  guide  its 
implementation.  These  three  components  each  focused  on  a  separate  aspect  of  expansion,  and  goals 
were  clarified  to  accompany  each  component: 

A.  Program  Expansion :  Expansion  of  the  YES  program  at  SLSC  by  an  additional  50  teens  and  related 
expansion  of  SLSC  staff 

a.  Expand  the  YES  program  to  reach  more  youth,  increase  diversity  of  the  youth,  and 
strengthen  STEM  content  focus 

B.  Stakeholder  Expansion :  Expansion  of  Outreach  to  the  Community  to  additional  high  school 
teachers  and  counselors,  scientists,  Navy  personnel  and  science  related  members  of  the 
community 

a.  Expand  the  SLSC  outreach  to  schools,  particularly  high  school  science  teachers  and 
school  counselors,  to  reach  more  students  and  teachers  and  to  develop  new  models  of 
outreach 

b.  Create  a  new  model  for  partnership  with  the  US  Navy  to  include  veterans  as  well  as 
active  duty  and  reserve  personnel  in  STEM  education  outreach  programming 

c.  Create  and  formalize  a  model  for  outreach  to  and  inclusion  of  members  of  the  science 
community  in  STEM  education  programming 

C.  National  Expansion:  Exploration  and  implementation  of  National  Expansion  initiatives  of  the  YES 
program 

a.  Improve  reflective  practice  of  all  educators  at  SLSC  to  create  a  cadre  of  leaders  for 
national  outreach  to  other  science  centers 

b.  Formalize  processes  and  collect  metrics  to  measure  YES  program  short-term  and  long¬ 
term  success 

c.  Codify  a  system  for  community  STEM  outreach  beyond  YES 

d.  Conduct  research  and  evaluation  to  provide  evidence  of  the  success  of  the  program  and 
to  identify  challenges 

e.  Create  a  strategic  plan,  resources  and  model  for  national  expansion  of  the  SLSC 
Community  STEM  Outreach  Program  and  begin  implementation 

Project  leaders  then  established  14  measureable  objectives  and/or  deliverables,  each  connecting 
directly  to  one  or  more  of  the  identified  goals.  These  objectives  provided  a  basis  to  measure  progress  in 
the  project's  implementation,  and  interim  reporting  cited  each  objective's  progress-to-date.  This  report 
presents  the  final  status  of  these  14  objectives  followed  by  an  analysis  of  these  results,  describing 
lessons  learned,  strengths  observed  and  program  enhancements  that  resulted  from  the  Community 
STEM  Outreach  Program. 
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Objectives 

Objective  1:  Increase  the  number  of  teens  participating  in  the  SLSC  YES  Program  by  50  (Status: 

Completed  and  Revised;  Goal  A.  a) 

The  YES  program  generally  enrolls  50  new  teens  at  the  beginning  of  the  spring  semester  each  year. 

In  February  2011,  the  YES  program  increased  new  teen  enrollment  to  100  to  evaluate  how  a  larger 
cohort  might  affect  operations.  These  teens  were  enrolled  through  existing  community  partner 
relationships  and  additional  partnerships  formed  during  this  project  (see  Objective  3). 

In  spring  2012,  the  YES  program  enrolled  a  smaller  cohort,  retaining  61  into  summer  programming. 
In  2013,  65  teens  continued  from  spring  orientation  into  summer  programming.  The  program 
anticipates  reducing  the  cohort  size  to  facilitate  individualized  attention  and  retention. 

Objective  2:  Increase  the  staff  size  beginning  to  support  the  increase  in  YES  teens,  community  outreach, 
and  other  expansion  efforts  (Status:  Initiated  and  Continuing;  Goal  A.a) 

ONR  funding  permitted  the  YES  program  to  increase  its  staff  support,  adding  an  Outreach  Manager, 
three  full-time  Senior  Educators,  and  three  part-time  Educators  in  January  2011.  With  these  newly  hired 
staff  members,  the  education  staff  represented  a  broader  range  of  specialties,  expanding  from  youth 
development  to  include  individuals  with  museum  studies  and  STEM  backgrounds. 

Facing  more  intensive  programming  requirements  in  the  summer  of  2011,  the  program  recruited  30 
additional  staff,  20  paid  summer  educator  interns  (all  certified  teachers  or  college  students)  and  10 
interns  to  document  the  program  for  use  in  multimedia  tools.  Paid  staff  also  relied  on  the  support  of 
unpaid  interns  from  Washington  University  in  St.  Louis  and  volunteers  from  local  community 
organizations. 

In  April  2013,  YES  founder  and  Chief  Educational  Outreach  Officer  Diane  Miller  accepted  a  new 
position  leading  education  efforts  at  The  Detroit  Zoo.  Siinya  Williams,  Senior  Director  of  Educational 
Outreach,  assumed  leadership  of  the  YES  program  bringing  more  than  20  years  of  experience  at  SLSC 
and  13  years  in  the  educational  outreach  department  itself.  Bert  Vescolani  assumed  the  role  of  principal 
investigator  for  the  Community  STEM  Outreach  Program.  Proper  notification  was  submitted  to  ONR 
representatives. 

At  the  end  of  2013,  excluding  teens,  the  YES  program  employed  three  administrative  staff,  including 
Williams  and  the  Outreach  Manager  as  well  as  an  administrative  manager.  The  program  also  employed 
five  full-time  Senior  Educators  and  four  part-time  Educators,  with  expertise  ranging  from  youth 
development  to  museum  studies  to  true  STEM  disciplines.  Senior  Educators  range  in  specialties  with  a 
majority  holding  Master's  level  degrees  in  STEM  fields,  youth  development,  or  museum  studies. 

Objective  3:  Increase  the  number  of  community  partnering  organizations  to  include  organizations  in  St. 
Louis  County  with  outreach  to  more  diverse  youth  (Status:  Initiated  and  Continuing;  Goal  A.a) 

The  YES  program  increased  the  number  of  community  partnerships  from  40  to  60  over  the  project 
period.  Highly-active  partners  (those  that  attend  more  than  75  percent  of  partner  meetings)  play  the 
greatest  roles  in  referring  teens  to  YES,  disseminating  YES  strategies  in  their  own  communities,  and 
advising  the  YES  program's  continued  development.  In  2010,  the  YES  program  engaged  25  highly-active 
partners.  In  2011,  YES  added  4  new  partners,  while  6  other  partners  either  lost  contact  or  struggled  to 
remain  in  operation.  In  2012  and  2013,  the  program  added  17  net  highly-active  partners,  bringing  the 
total  to  40.  With  a  focus  on  diversity,  these  new  partners  included  the  International  Studies  Program  at 
Saint  Louis  Public  Schools,  Harris-Stowe  State  University,  Mission:  St.  Louis,  Upward  Bound,  and  St.  Louis 
Language  Immersion  Schools. 

These  partnerships  resulted  in  an  increased  range  of  diversity  within  the  YES  program.  In  2011,  97 
percent  of  teens  in  the  YES  program  identified  as  African  American,  with  individuals  identifying  as 
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Caucasian,  Latino(a)/Hispanic  American,  and  Other  each  comprising  an  additional  1  percent.  In  2013, 
program  demographics  reflected  greater  diversity,  at  84  percent  African  American,  4  percent  Caucasian, 
3  percent  Asian/Pacific  Islander,  1  percent  Hispanic/Latino(a),  2  percent  Other,  and  6  percent  Not 
Identified.  The  cultural  and  national  identities  represented  in  YES  add  further  diversity,  with  students 
identifying  more  specifically  as  Nigerian,  Ethiopian,  Ogoni-African,  Bosnian  and  Nepali. 

YES  community  partners  attend  monthly  meetings  for  professional  development.  Here,  they  gain 
exposure  to  out-of-school  STEM  learning  activities  and  access  exclusive  curricular  materials  to  use  with 
youth  in  their  programs.  On  average,  35  partners  attend  these  monthly  meetings. 

Objective  4:  Reach  new  school  audiences  with  existing  and  new  SLSC  programming ,  focusing  on  school 
counselors  and  high  school  science  teachers  (Initiated  and  Continuing:  Goal  B.a) 

In  September  2011,  the  YES  program  took  part  in  an  organization-wide  direct  marketing  campaign 
for  schools  in  the  St.  Louis  metropolitan  area.  The  program  leveraged  these  organizational  efforts  to 
reach  new  school  audiences  rather  than  approach  the  same  partners  individually.  Since  the  inception  of 
the  direct  marketing  campaign  through  the  present,  teachers  have  been  invited  to  attend  all 
OMNIMAX®  film  openings.  Science  Cafes,  and  Special  Exhibit  openings  at  SLSC.  The  educator  mailing  list 
currently  includes  contact  information  for  12,815  educators  in  the  St.  Louis  region.  Approximately  900 
educators  attended  educator  preview  events  in  2013.  In  addition,  educators  have  been  invited  to  a 
special  Science  Maker  event  in  2012  and  Minority  Scientists  Showcase,  an  event  hosted  annually  on  the 
weekend  of  Martin  Luther  King,  Jr.  Day  (average  total  attendance,  2011-2013:  4,377),  which  includes  an 
YES  teens  demonstrating  activities  alongside  SLSC  educators  and  20-30  special  guest  vendors.  An 
average  of  110  YES  teens  participated  in  Minority  Scientists  Showcase  each  year  from  2011  to  2013. 
These  marketing  efforts  also  provided  the  YES  program  with  great  returns  in  the  number  of  community 
organization  partnerships  (see  Objective  3). 

Programs  throughout  the  educational  outreach  department  (beyond  YES)  from  2011  through  2013 
focused  on  creating  or  strengthening  partnerships  with  local  schools.  For  example,  high  school  biology 
teachers  were  invited  to  prepare  students  to  compete  in  the  St.  Louis  regional  Brain  Bee  hosted  at  SLSC 
in  collaboration  with  Dr.  Erik  Herzog  of  Washington  University  in  St.  Louis.  Family  Math  educators 
hosted  events  at  ten  area  schools  and  four  community  partner  locations  to  involve  parents  and  children 
in  mathematics  activities.  While  at  these  locations,  SLSC  staff  worked  with  educators  to  describe  more 
programming  available  through  the  educational  outreach  department. 

Objective  5:  Strengthen  the  STEM  content  focus  of  YES  components  to  include  stronger  emphasis  on 
STEM  in  existing  components  and  addition  of  new  components  with  content  relevant  to  the  US  Navy 
(Status:  Initiated  and  Continuing;  Goal  A.  a) 

New  YES  components  with  enhanced  STEM  foci  were  introduced  in  2011-2012  and  included  in  the 
interim  report  in  March  2012.  Stronger  STEM  content  arose  from  consultation  with  and  guidance  of 
scientists  and  professionals,  including  Dr.  Erik  Herzog  of  Washington  University  in  St.  Louis,  Gibron 
Burchett  of  HOSCO  Foods,  and  RADM  Lee  Metcalf  of  the  United  States  Navy  (retired).  These  advisors 
recommended  curricular  additions,  participated  in  planning,  and,  at  times,  assisted  with  program 
activities.  These  components  have  largely  remained  consistent  through  the  end  of  2013.  Components  in 
fall  2013  included  Neuroscience,  Astronomy,  Robotics  &  Engineering,  Chemistry  &  Biotechnology,  and 
Science  Corner.  Science  Corner  combines  elements  of  the  Plant  Biochemistry  and  Agriscience 
components  from  the  prior  year,  inviting  students  to  participate  in  sustainable  agriculture  practices  on  a 
green  lot  near  the  Taylor  Community  Science  Resource  Center.  Biofuels  &  Energy  is  no  longer  offered  as 
a  component,  though  elements  have  been  incorporated  into  the  Chemistry  &  Biotechnology 
component. 

•  Neuroscience  -  Teens  explore  the  function  and  impact  of  processes  in  the  human  brain. 
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Teens  investigated  neuronal  anatomy  and  communication,  using  project-based  learning  to  work 
through  the  steps  of  the  scientific  method.  Teens  may  prepare  for  competition  in  the  regional  Brain 
Bee  competition. 

•  Astronomy  -  Teens  gain  an  understanding  of  how  human  understanding  of  astronomy  as  a 
science  has  changed  over  time,  learning  about  important  scientific  influencers  and  technological 
advances  that  changed  human  understanding  of  our  place  in  the  universe.  These  discussions 
integrate  STEM  concepts  like  astronomical  scale,  light,  optics,  and  force  and  motion. 

•  Design  Engineering  -  Focused  on  understanding  the  engineering  design  process,  the 
robotics  component  engages  teens  in  activities  involving  NXT  Mindstorm  kits.  Snap  Circuits  and 
Arduinos.  In  past  years,  teens  have  competed  in  the  FIRST  Tech  Challenge  and  FIRST  Robotics 
Competition. 

•  Technology  -  Through  a  special  partnership  with  Girls  RISENet,  UrbanFUTURE,  and  Arete, 
ten  girls  from  underrepresented  backgrounds  will  learn  computer  programming  and  then  share  their 
new  knowledge  with  children  from  community  partners.  Professionals  from  Arete,  specializing  in 
STEM  instruction  for  girls,  will  teach  this  girls-only  component,  introducing  computer  programming 
through  Scratch,  a  program  produced  by  the  MIT  Media  Lab. 

•  Chemistry  &  Biotechnology  -  Teens  learn  about  matter  and  how  properties  of  materials  are 
determined  by  their  molecular  makeup.  Teens  conduct  their  own  experiments  with  matter  to 
understand  more  about  the  elements  of  the  periodic  table  and  how  they  react  based  on  their 
atomic  structure.  Going  forward,  this  component  will  also  focus  on  biotechnology,  examining  how 
chemistry  impacts  how  we  study  living  things  and  how  chemical  properties  can  play  a  role  in 
developing  new  biotechnology  tools. 

•  Science  Corner  -  Teens  go  beyond  the  basics  of  plant  science,  exploring  how  the  challenges 
facing  the  global  food  supply  may  be  managed  through  alternative  agriculture  practices.  Working 
with  local  entrepreneurs,  teens  have  constructed  aquaponic  and  aeroponic  systems  to  grow  plants 
in  the  Taylor  Community  Science  Resource  Center.  Teens  also  engage  in  agriscience  activities  on 
Science  Corner,  a  green  lot  located  near  the  Taylor  Center. 

Objective  6:  Develop  new  opportunities  for  partnership  between  SLSC  staff  and  Navy  personnel  to 
support  the  YES  Program  and  other  SLSC  outreach  activities  (Status:  Initiated  and  Revised;  Goal  B.b) 

Diane  Miller  and  Retired  Navy  RADM  Lee  Metcalf  established  the  St.  Louis  Regional  STEM  Coalition, 
with  members  representing  a  broad  constituency  including  Southern  Illinois  University  -  Edwardsville, 
Washington  University  in  St.  Louis,  Cortex  Innovation  Community,  University  of  Missouri  -  Columbia, 
and  University  of  Missouri  -  St.  Louis  alongside  the  Saint  Louis  Science  Center.  The  Coalition  aimed  to 
increase  the  number  of  local  STEM  programs,  align  funding  with  needed  initiatives  and  increase  the 
number  of  available  employees  possessing  varied  STEM  competencies,  with  the  ultimate  goal  of 
enhancing  the  local  workforce  and  its  economic  impact  on  the  St.  Louis  region.  The  Coalition  compiled  a 
resource  binder  that  detailed  the  need  to  collect  academic  achievement  data  across  a  broad  range  of 
organizations  throughout  the  St.  Louis  region.  This  material  informed  a  Coalition-created  proposal  to 
fund  the  creation  of  that  database.  At  the  end  of  the  Community  STEM  Outreach  Program,  funding  for 
this  initiative  had  not  been  granted,  though  efforts  to  establish  the  regional  database  or  a  modified 
system  will  continue. 

RADM  Lee  Metcalf  advises  SLSC  as  a  trustee  and  played  a  major  role  in  guiding  the  revisions  of  STEM 
content  and  curricular  materials  in  each  component.  The  YES  program  experienced  difficulties  creating 
consistent  opportunities  to  engage  Navy  personnel.  One  of  the  main  tenets  of  YES  is  stability  -  teens 
rely  on  program  staff  to  be  consistent.  Educators  model  dependability  through  regular,  reliable 
attendance  at  the  program.  Navy  personnel  are  necessarily  more  transient.  Faced  with  other  critical 
responsibilities,  they  are  unable  to  commit  to  this  consistent  attendance.  Considering  these  factors,  the 
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YES  program  found  success  engaging  Navy  personnel  in  shorter-term  projects  like  curriculum 
development,  capital  improvements,  or  intermittent  special  events.  YES  leaders  strongly  advocate 
engaging  Navy  personnel  through  these  and  similar  episodic  opportunities.  RADM  Metcalf's  involvement 
connected  Science  Center  President  Bert  Vescolani  to  several  opportunities  with  the  Navy  and  other 
military  entities.  In  the  last  year,  Vescolani  attended  an  Employer  Support  of  the  Guard  and  Reserve 
(ESGR)  event  in  San  Diego,  touring  the  commissioned  Los  Angeles-class  USS  Jefferson  City  submarine. 

His  experience  there,  and  his  connection  with  RADM  Metcalf,  brought  Vescolani  to  speak  at  the  Navy 
Ball  in  St.  Louis,  sharing  the  importance  of  STEM  academic  and  work  force  development  with  attendees 
(see  Speaking  Engagements  and  Articles  below). 

In  addition  to  Navy  personnel,  the  program  also  enrolled  two  Sea  Cadet  students  in  January  2012 
and  twelve  students  from  Navy-linked  Cleveland  NJROTC  High  School  completed  the  program  in  during 
the  funding  period.  The  program  promoted  the  Naval  Summer  STEM  Program  to  its  teens  each  year, 
with  one  attending  recently  in  summer  2013.  A  number  of  YES  teens  graduating  in  2012  and  2013 
elected  to  pursue  careers  in  the  Navy  or  other  military  outlets. 

Objective  7:  Develop  and  formalize  opportunities  for  involvement  of  practicing  and  retired  scientists  in 
the  community  (Status:  Initiated  and  Continuing;  Goal  B.c) 

During  the  Community  STEM  Outreach  Program  period,  the  YES  program  engaged  practicing 
scientists  in  each  component  to  help  develop  STEM  content  and  to  participate  in  learning  labs  either  in 
person  or  via  electronic  media  like  Skype  or  Second  Life.  YES  educators,  per  their  job  descriptions,  were 
encouraged  to  seek  individual  partnerships  with  their  science  network.  With  many  educators  joining  the 
program  having  recently  completed  their  education  at  nearby  universities,  they  were  poised  to  engage 
the  scientist  population. 

Dr.  Erik  Herzog  of  Washington  University  in  St.  Louis  (WUSTL)  worked  with  the  YES  Senior  Educator 
in  the  Neuroscience  component  to  coordinate  the  St.  Louis  regional  Brain  Bee.  Dr.  Parag  Banerjee,  also 
from  WUSTL,  began  volunteering  with  the  YES  program  in  spring  2012  and  engaged  teens  in 
nanotechnology  lab  activities  on  the  university's  campus  in  the  summers  of  2012  and  2013.  Dr.  Lisa 
Schechter,  Assistant  Professor  at  the  University  of  Missouri  -  St.  Louis,  and  doctoral  students  from  her 
lab  engaged  YES  teens  in  activities  involving  plant-microbe  interactions.  YES  teens  competed  in  the  Bell 
Agriscience  competition  sponsored  by  Dr.  Ellis  Bell,  Monsanto,  the  Soy  Bean  Association,  and 
AgriBusiness  Organization.  From  their  involvement  eight  YES  teens  received  scholarship  awards. 
Additional  scientists  teamed  with  teens  to  present  Science  Cafes:  Dr.  Terri  Rebmann  (Saint  Louis 
University),  Dr.  Dann  Moran  (WUSTL),  Dr.  Jason  Wagner  (WUSTL),  Dr.  Stephen  Blake  (WUSTL),  Dr.  Nicole 
Gugliucci  (Southern  Illinois  University  -  Edwardsville,  SIUE),  Dr.  Glenn  Morrison  (Missouri  University  of 
Science  and  Technology),  and  Dr.  Jack  Glassman  (SIUE).  These  science  cafes  brought  teens  and  scientists 
together  in  an  informal  environment  to  discuss  and  learn  about  STEM  issues.  Scientists  and 
professionals  also  participated  in  special  events  such  as  student  poster  demonstrations, 
entrepreneurship  review  panels,  and  YES's  professional  networking  day,  which  provides  the  teens  with 
an  arena  in  which  to  practice  and  develop  networking  skills  through  interactions  with  volunteers  from 
the  community. 

Objective  8:  Strengthen  the  reflective  practice  of  YES  staff  and  SLSC  educators  through  additional 
training  and  ongoing  support  (Status:  Initiated  and  Continuing;  Goal  C.a) 

In  fall  2012,  the  YES  program  committed  to  weekly  professional  development  activities  for 
education  staff.  In  addition  to  more  recognizable  forms  of  professional  development,  YES  educators 
also  engage  in  group  reflective  practice  after  each  program  session,  weekly  on  Saturdays  during  the 
school  year  and  daily  during  summer  programming.  These  reflective  meetings  illuminate  challenges  that 
educators  may  be  facing,  informing  programmatic  changes  and  ongoing  development.  Educators  also 
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build  a  stronger  sense  of  camaraderie,  creating  a  community  in  which  staff  members  are  better 
equipped  to  support  each  other. 

Objective  9:  Train  and  support  a  cadre  of  STEM  education  leaders  who  can  train  others  in  effective 
strategies  to  build  programs  that  are  community  relevant,  youth  development  focused,  and  strong  in 
STEM  content  (Status:  Initiated  and  Continuing;  Goal  C.a) 

YES  leaders  and  educators  instituted  a  "train  the  trainer"  model  along  several  dimensions  in  order  to 
widen  the  reach  of  the  program.  YES  staff  train  individuals  to  share  STEM  content  with  others,  rather 
than  focusing  solely  on  directly  conveying  STEM  content.  In  this  way,  the  program  creates  a  network 
effect:  an  individual  learns  how  to  present  an  activity  at  the  YES  program  and  then  shares  that  activity 
with  30  children  at  her  own  organization,  or  a  YES  teen  develops  a  STEM  activity  with  guidance  from  a 
Senior  Educator  and  then  shares  that  activity  with  1,000  younger  children  over  the  course  of  a  summer. 

YES  implemented  this  system  with  three  primary  audiences:  YES  educators  themselves,  the  YES 
teens,  and  community  partner  leaders.  Educator  training  and  professional  development  promoted  the 
importance  of  coaching  to  empower  teens  to  lead  activities  and  educate  others.  In  the  YES  components, 
Senior  Educators  helped  teens  learn  to  present  existing  STEM-focused  activities  or  guided  teens  as  they 
created  their  own.  Teens  then  presented  these  activities  to  younger  children  who  visited  the  Taylor 
Community  Science  Resource  Center  for  summer  field  trips.  Other  teens  presented  activities  to  visiting 
families  and  children  at  SLSC. 

YES  also  focused  on  training  leaders  from  its  community  partner  organizations.  All  community 
partners  are  invited  to  attend  a  monthly  session  that  strongly  focuses  on  integrating  STEM  activities  into 
a  wide  variety  of  programs.  Each  month,  these  community  partners  participate  in  a  STEM  learning 
activity  and  may  receive  the  materials  they  need  to  complete  the  activity  with  the  youth  and  children  at 
their  organizations.  On  average,  more  than  30  partners  attend  these  meetings. 

Objective  10:  Create  electronic,  multimedia  documentation  of  all  Community  STEM  Outreach  activities 
and  staff  reflections  for  support  of  expansion  efforts  (Status:  Completed;  Goal  C.e) 

External  evaluator  Christine  (Kit)  Klein  of  Klein  Consulting  completed  documentation  of  the  YES 
program,  preliminarily  available  as  both  a  web  resource  (beta  site  in  development 
at  www.yescirclesofsupport.com)  and  a  printable  PDF  document  (Appendix 

A,  http://media.wix.com/ugd/34fb60  9eccflefe470f60f57e4db06f3446fec.pdf).  This  report  reviews  the 
critical  elements  of  YES,  using  the  program's  logic  model  as  the  basis  for  organization.  These  elements 
were  identified  through  Klein's  work  in  evaluation  throughout  the  project  (see  Objective  11).  As 
opposed  to  assuming  broad  applicability  of  the  YES  model,  the  multimedia  tool  leads  users  through  a 
discussion  and  thought  process.  This  process  guides  the  user  to  develop  a  STEM  youth  program  suitable 
to  meet  the  needs  and  leverage  the  strengths  of  the  community  being  served. 

Objective  11:  Conduct  evaluation  to  support  and  provide  evidence  of  the  success  of  the  program  and  to 
identify  challenges  (Status:  Initiated,  Completed,  and  Continuing;  Goal  C.b,  C.d) 

Both  internal  and  external  evaluators  supported  the  operations  of  the  YES  program  throughout  the 
implementation  of  the  Community  STEM  Outreach  Program.  Internal  evaluators,  led  by  Associate 
Director  of  Research  and  Evaluation  Elisa  Israel,  conducted  brief  impact  surveys  that  provided  feedback 
on  how  participants  perceived  the  impact  of  their  experience  with  YES.  This  system,  called  SAMI  (System 
for  Assessing  Mission  Impact)  asks  participants  to  rate  their  experience  as  it  relates  to  STEM 
engagement  along  four  dimensions:  knowledge,  enjoyment,  interest,  and  attitude  change.  A  rating  of 
"1"  indicates  that  a  specific  dimension  was  "Not  at  All"  enhanced.  A  rating  of  "4"  indicates  that  the 
dimension  was  enhanced  "Quite  a  Bit."  This  system  is  primarily  used  to  monitor  the  progress  of 
programs  internally  and  is  not  intended  to  provide  external  validity.  Internally,  SLSC  defines  program 
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impact  using  a  scale  from  low  impact  (<10.99  out  of  16)  to  Very  High  Impact  (>15.00  out  of  16). 
Throughout  the  Community  STEM  Outreach  Program  period  from  2011  to  2013,  YES  teens  reported  an 
average  impact  score  of  13.18,  defined  as  "High  Impact." 

External  evaluator  Klein  Consulting  developed  and  tracked  evaluative  measures  to  determine  more 
specific  impacts  of  the  program.  These  metrics  provide  more  meaning  for  an  external  audience.  Klein 
first  focused  on  defining  and  making  salient  the  most  critical  elements  of  YES,  as  reported  in  evaluation 
reports  submitted  in  the  first  two  years  of  the  program.  Later,  Klein  transitioned  to  discussions  of  how 
the  YES  program  may  translate  to  other  organizations.  In  her  final  evaluation  report  (Appendix  B),  Klein 
provides  a  summative  overview  of  the  YES  program,  its  impact  on  underserved  teens  in  the  St.  Louis 
Metropolitan  Statistical  Area,  and  guiding  factors  for  translating  elements  of  the  program  into  a  national 
expansion  effort.  This  work  culminated  in  the  Circles  of  Support  multimedia  tool  (see  Objective  10). 

Since  the  end  of  the  Community  STEM  Outreach  Program  period,  YES  program  leaders  have  taken 
steps  to  ensure  continued  data  collection  and  to  refine  evaluation  efforts  going  forward.  Meeting  with 
representatives  from  the  grants  and  evaluation  departments  at  the  Science  Center,  YES  educators 
prepared  an  interim  evaluation  system  to  collect  data  used  in  grants  proposals  and  other  fund  raising 
initiatives.  This  interim  system  is  being  implemented  as  of  spring  2014  to  ensure  the  next  chapters  of 
the  YES  story  can  be  supported  by  data.  Meanwhile,  three  YES  Senior  Educators  have  formed  a  task 
force  to  1)  identify  and  clarify  the  goals  of  the  YES  program,  2)  determine  objectives  and  measurable 
outcomes  consistent  with  those  goals,  and  3)  devise  a  system  to  collect  data  related  to  those  objectives 
and  outcomes.  As  of  late  February  2014,  the  team  of  educators  had  amassed  documents  that  showed 
various  representations  of  YES  goals  throughout  the  program's  16  year  history.  From  these  documents, 
the  educator  team  identified  three  key  goals  and  related  objectives.  Put  simply,  the  YES  program  should: 
prepare  teens  for  work,  prepare  teens  to  pursue  higher  education,  and  prepare  teens  to  appreciate 
science.  Pending  approval  from  YES  and  Science  Center  leadership,  the  team  will  next  identify 
measureable  outcomes  for  the  goals  and  objectives  identified  and  then  propose  a  plan  to  collect  data 
matching  those  outcomes.  This  plan  will  continue  to  evolve  through  spring  2014. 

Objective  12:  Identify  research  questions  related  to  the  Community  STEM  Outreach,  and  create 
strategies  for  moving  forward  with  that  research,  including  seeking  additional  funding  for  such  research 
(Status:  Completed;  Goal  C.d) 

Klein  Consulting  was  also  contracted  to  synthesize  research  available  on  STEM  youth  programs,  add 
its  own  evaluation  on  YES  to  this  work,  and  identify  ongoing  topics  for  research  (see  Appendix  C, 
Proposed  Research  Agenda).  Klein  advocated  for  further  research,  but  using  new  methods,  on  the 
research  questions  she  identified  during  the  project  regarding  the  YES  program's  short-  and  long-term 
impacts.  She  also  recommended  additional  effort  to  research  and/or  evaluate  the  long-term  impacts  of 
the  program  on  alumni,  through  continued  tracking  and  monitoring  of  outcomes. 

Additional  efforts  in  YES  evaluation  have  been  proposed  but  not  funded.  SLSC  coordinated  with  a 
wide  variety  of  organizations  to  submit  a  proposal  to  fund  an  investigation  into  the  human  experience, 
from  lenses  including  primatology,  anthropology,  psychology,  and  youth  development.  This  proposal 
and  others  are  also  described  in  Appendix  C.  Following  the  ONR  funding  end  date  on  December  31, 

2013,  SLSC  and  Klein  Consulting  submitted  an  NSF  RAPID  proposal  to  evaluate  the  effect  of  leadership 
change  throughout  the  period  of  ONR  funding. 

Objective  13:  Create  a  strategic  plan  for  national  expansion  of  the  SLSC  Community  STEM  Outreach 
Program  (Status:  Initiated  and  Revised:  Goal  C.c,  C.e) 

The  YES  program  hosted  the  Community  STEM  National  Collaborative  Conference  in  April  2013, 
including  organizational  and  program  leaders  from  several  informal  STEM  institutions,  including:  Boston 
Children's  Museum,  Museum  of  Life  and  Science,  Bishop  Museum,  Center  for  Lifelong  Science  Learning, 
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Oregon  Museum  of  Science  and  Industry,  Franklin  Institute,  University  of  California-Berkeley,  Carnegie 
Museum  of  Natural  History,  California  Science  Center,  and  Pacific  Science  Center.  The  event  also 
welcomed  Joseph  Cohn,  CDR  &  Military  Deputy,  Human  and  Bioengineered  Systems  Division,  Office  of 
Naval  Research. 

The  group  discussed  the  critical  elements  for  national  expansion,  conceptualizing  a  three-pronged 
program  structure  focused  on  youth  development,  workforce  preparation,  and  STEM  content.  The 
group  assessed  these  three  elements  as  critical  in  any  attempt  to  implement  YES-based  programming. 
The  group  ended  the  session  with  a  charge  to  continue  research  and  discussion  via  an  online 
community.  This  community  was  established  and  10  conference  attendees  joined  the  group.  Members 
provided  information  on  programs  at  their  organizations  similar  to  YES.  Future  efforts  may  bring  the 
collaborative  partners  back  together  in  the  coming  years  to  establish  a  more  formalized  plan  for  national 
expansion. 

Klein  Consulting  was  also  independently  charged  with  creating  a  strategic  plan  for  YES 
dissemination.  Klein's  plan  strongly  emphasizes  the  dissemination  of  the  Circles  of  Support  multimedia 
tool  (see  Objective  10)  and  professional  development  for  informal  STEM  educators  based  on  this  tool. 
See  Appendix  D  for  a  review  of  Klein's  plan  for  Proposed  Dissemination  of  YES. 

Objective  14:  Identify  resources,  including  national  partners,  for  national  outreach  (Status:  Initiated  and 
Revised;  Goal  C.e) 

Klein  Consulting's  Circles  of  Support  provides  a  lasting  resource  that  educators  can  use  to  guide 
development  of  programs  similar  to  YES.  The  tool  is  open-ended  to  allow  thoughtful  discussion  on  what 
elements  of  programming  will  be  most  important  for  specific  community  needs.  Currently,  YES 
educators  are  engaged  in  reviewing  and  revising  the  tool  with  updated  curricula  and  standards.  Once 
complete,  the  product  will  be  advertised  through  a  variety  of  informal  STEM  professional  networks 
including  the  attendees  of  the  Community  STEM  National  Collaborative  Conference  (see  Objective  13). 
This  collaboration  will  serve  as  the  first  phase  of  expansion  and  dissemination  of  the  Circles  of  Support 
tool.  Partners  include  the  same  as  those  in  Objective  13:  Boston  Children's  Museum,  Museum  of  Life 
and  Science,  Bishop  Museum,  Center  for  Lifelong  Science  Learning,  Oregon  Museum  of  Science  and 
Industry,  Franklin  Institute,  University  of  California-Berkeley,  Carnegie  Museum  of  Natural  History, 
California  Science  Center,  and  Pacific  Science  Center. 

Analysis:  Lessons  Learned  and  Program  Enhancements 

The  Community  STEM  Outreach  Program  provided  an  arena  in  which  the  YES  program  could 
transition  into  a  phase  of  organizational  development  focusing  on  sustainability.  Having  grown  from 
serving  a  mere  handful  of  teens  when  it  began  in  1997,  YES  now  engages  more  than  200  teens  in  STEM 
programming  each  year.  With  support  from  ONR,  the  YES  program  investigated  several  options  for 
enhancing  sustainability.  Moreover,  through  strengthened  relationships  with  the  Navy  and  several  other 
organizations,  YES  program  leaders  identified  a  broader  range  of  partners  that  can  contribute  to  teens' 
post-secondary  success.  As  a  result,  YES  now  engages  a  wide  range  of  post-secondary  institutions, 
including  military  entities,  technical  and  trade  schools,  and  a  wider  selection  of  two-  and  four-year 
colleges. 

Catalyzing  Change 

With  thirteen  years  of  operation  by  2010,  the  YES  program  was  approaching  a  new  phase  in 
organizational  development.  Having  grown  significantly  in  its  first  decade  of  programming,  the  YES 
program  needed  opportunities  to  test  new  forms  of  sustainability,  operations,  and  outreach.  Support 
from  ONR  provided  these  opportunities,  catalyzing  a  transition  to  enhance  program  sustainability. 
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Several  aspects  of  the  YES  program  have  incorporated  new  elements  or  strengthened  existing  elements 
to  improve  sustainability. 

In  2013,  the  YES  program  augmented  its  existing  internship  program.  In  prior  years,  teens  finishing 
their  junior  or  senior  years  would  enter  internships  at  the  Science  Center  or  at  a  variety  of  community 
organizations  and  businesses,  while  remaining  on  the  Science  Center's  payroll.  These  internships 
enhance  work  skills  and  challenge  teens  to  adapt  to  a  new  "system"  of  work  place  culture,  values,  and 
responsibilities.  In  2013,  the  Science  Center  solicited  support  from  its  community  partners  under  a  new 
internship  system.  The  Science  Center  continues  to  pay  students  who  intern  during  the  summer 
following  their  junior  year.  However,  following  their  senior  year,  teens  return  to  the  same  community 
partner  as  a  directly  employed  intern  of  that  organization.  Through  all  of  these  internships,  teens 
continue  to  receive  the  benefits  of  internship  experiences,  while  the  Science  Center  shares  the 
substantial  financial  requirement  of  the  internship  program  across  several  partner  organizations.  YES 
educators  and  staff  will  continue  to  network  with  community  partners  and  businesses  to  expand  these 
opportunities  going  forward.  Additionally,  ten  YES  teens  will  intern  in  one  of  a  select  few  departments 
within  the  Science  Center  itself.  These  teens  will  gain  a  greater  understanding  of  the  Science  Center's 
organizational  system,  while  becoming  advocates  for  the  Science  Center  and  contributing  to  its  work. 

Since  the  Community  STEM  Outreach  Program  period,  YES  administrative  staff  have  also  teamed 
with  the  Science  Center's  volunteer  resources  department  to  leverage  processes  across  departments  to 
improve  overall  efficiency.  Using  the  organization's  standard  volunteer  recruitment  and  retention 
procedures,  the  YES  program  has  enhanced  its  volunteer  experience  without  substantial  increases  in  YES 
personnel  responsibility. 

The  YES  program  also  enhanced  its  use  of  community  resources  and  focused  its  efforts  on  specific 
segments  of  the  broad  "underrepresented  population"  to  achieve  greater  impact  and  address  current 
concerns  in  STEM  recruitment.  The  YES  program's  newly  introduced  Technology  component  will  enroll 
ten  girls  from  UrbanFUTURE,  a  program  specializing  in  6th  through  8th  grade  interventions.  Transitioning 
from  middle  school  to  high  school,  and  from  UrbanFUTURE  to  YES,  these  girls  will  continue  to  receive 
support  critical  for  academic  success.  This  enrollment  pipeline  and  the  girls-only  component  structure 
are  both  departures  from  the  standard  YES  model,  and  both  have  potential  to  enhance  program 
outcomes.  The  effects  of  these  changes  will  be  reviewed  over  the  coming  years. 

In  line  with  this  review,  evaluation  efforts  in  the  YES  program  have  been  revised  and  continue  to  be 
improved.  In  the  current  interim  period,  educators  will  continue  to  collect  several  elements  of  data 
collected  by  Klein  Consulting  during  the  Community  STEM  Outreach  Program  period.  These  activities  will 
provide  continuing  documentation  of  program  outcomes  important  for  sharing  the  YES  story.  Three  YES 
Senior  Educators  are  also  working  on  a  broader  project  to  revise  and  improve  evaluation  procedures 
going  forward  (see  Objective  11). 

The  YES  program  has  also  instituted  stronger  attendance  documentation  and  procedures  to  enhance 
the  YES  story.  The  original  "Deliberate  Design"  of  the  program  held  that  all  educators  should 
communicate  with  their  teens  at  least  once  each  week,  whether  in  programming  or  by  phone.  This 
element  has  been  reinstated,  and  the  program  manager  collects  records  of  each  educator's 
communication  with  teens.  Meanwhile,  new  attendance  policies  have  reduced  the  effort  of  educators  in 
these  communications.  Teens  who  are  not  able  to  attend  sessions  during  the  school  year  must 
document  their  reasons  for  absence  in  order  to  continue  participation  during  the  summer.  This 
enhances  accountability  and  incentivizes  students  to  contact  their  educator  proactively. 


Redefining  Success 

The  Community  STEM  Outreach  Program  significantly  enhanced  YES  philosophy.  Prior  to  2010,  the 
primary  goal  for  each  YES  teen  was  one-dimensional:  each  teen  should  attend  a  four-year  college 
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following  high  school  graduation.  Throughout  the  three-year  project,  this  goal  became  increasingly 
better  defined.  RADM  Lee  Metcalf  attended  networking  events  and  held  roundtable  conversations, 
offering  a  different  perspective  on  Navy  and  military  recruitment.  Beyond  the  Navy,  the  YES  program 
also  reached  out  to  a  broader  range  of  education  partners,  including  Ranken  Technical  College  and 
Harris-Stowe  State  University.  Involving  this  expanded  group  of  partners  provided  YES  teens  with 
exposure  to  a  wider  range  of  post-secondary  options. 

This  new  philosophy  affected  other  elements  of  programming  as  well.  The  YES  program  more 
accurately  recognized  the  unique  value  offered  by  post-secondary  options  beyond  standard  four-year 
education.  The  inherent  structure  and  community  found  in  military  service  create  environments  that 
promote  success  for  many  young  teens  out  of  high  school.  Technical  schools  may  capitalize  on  more 
focused  teen  interests,  and  job  placement  for  technical  careers  is  exceptional. 

YES  educators  and  staff  also  began  to  pay  attention  to  a  broader  range  of  skills  that  play  into  this 
wider  range  of  post-secondary  options.  Accordingly,  they  expanded  their  searches  for  training  and 
development  opportunities.  College  Preparation  sessions  expanded  into  College  and  Career  Readiness, 
tailoring  senior  programming  to  focus  on  each  teen's  expressed  post-secondary  goal. 

Structuring  Improvement 

Support  from  ONR  provided  an  opportunity  to  modify  and  refine  YES  infrastructure,  enhance 
program  evaluation  efforts,  and  diversify  YES  funding  sources.  Combined,  these  elements  presented  an 
opportunity  to  enrich  the  narrative  of  the  YES  program.  Evaluation  data  demonstrating  150  percent 
improvement  in  post-secondary  matriculation,  and  more  accurate  demographic  and  participant 
information  provides  a  strong  foundation  for  future  funding  proposals  and  partnership  requests.  This 
evaluation  data  also  informed  and  improved  YES  programming,  and  existing  funders  are  more  and  more 
satisfied  with  outcomes  reported  to  them.  Ultimately,  the  Science  Center  now  possesses  a  truly 
compelling  story  to  tell  when  asked  about  YES. 

Going  forward,  the  Science  Center  sees  great  potential  in  applying  a  similar  process  to  additional 
programs.  Solidifying  a  process  to  create  exciting  narratives  driven  by  solid  data  and  superior 
programming  is  an  exciting  prospect  for  many  organizations  and  businesses.  Science  Center  leaders  are 
eager  to  test  this  newly  defined  process  on  other  programs  to  elicit  more  compelling  stories  and 
strengthen  already  powerful  programming. 

Analysis:  Strengths  and  Challenges 

Work  toward  the  objectives  described  above  presented  several  strengths  and  challenges  during 
implementation.  Each  of  the  three  broad  components  of  expansion  promoted  by  the  Community  STEM 
Outreach  Program  achieved  differing  levels  of  success. 

Programmatic  Expansion 

With  regards  to  programmatic  expansion,  the  YES  program  increased  enrollment  by  50  teens  in 
2011,  expanded  the  range  of  diversity  among  its  participants,  and  tailored  components  to  focus  more 
strongly  on  STEM  content.  Enrollment  expansion  created  challenges  with  participant  retention  and 
impact.  These  factors  countered  the  positive  impact  of  enrolling  new  students.  As  a  result,  program 
educators  and  leaders  advocated  for  a  return  to  standard  cohort  enrollment.  Meanwhile,  enhancing  the 
diversity  of  participants  and  strengthening  STEM  content  contributed  positively  to  the  teens'  identity 
development.  More  than  seven  of  every  ten  teens  ended  up  naming  a  STEM  career  in  their  interest 
inventory.  YES  provided  an  opportunity  for  its  teens  to  see  themselves  as  scientists  by  engaging  them 
firsthand  in  scientific  inquiry,  and  their  career  interests  followed  suit. 

Several  potential  factors  may  have  contributed  to  the  challenges  faced  when  increasing  enrollment. 
While  staff  size  was  increased  at  the  same  time,  the  program  also  relied  on  several  volunteers. 
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Volunteers  frequently  have  schedule  conflicts  that  prevent  them  from  maintaining  a  steady  routine,  and 
teens  were  not  able  to  rely  on  the  consistency  demonstrated  by  paid  staff.  Furthermore,  the  space 
allocated  to  YES  was  fixed.  Having  maximized  classroom  space  already,  the  program  could  not  expand 
further  and  was  resigned  to  increasing  classroom  group  size.  Even  with  the  added  support  offered  by 
additional  staff,  interns,  and  volunteers,  the  Senior  Education  staff  could  not  provide  the  same  level  of 
individualized  support  that  classically  defined  YES  programming.  As  a  result,  program  educators 
suggested  that  teens  felt  less  connected  to  the  program  and,  as  a  result,  attended  less  frequently. 

Meanwhile,  the  program  identified  several  strengths  in  involving  volunteers,  particularly  science 
researchers  or  Navy  professionals,  in  curriculum  development  and  special  events.  For  these  volunteers, 
it  proved  difficult  to  commit  to  a  regular  schedule,  and  it  would  be  detrimental  to  the  teens' 
involvement  in  the  program  to  promise  reliable  attendance  and  fail  to  follow  through.  For  special  one¬ 
time  or  short-term  projects,  such  as  FIRST  Robotics  competitions  or  the  Science  Center's  NeuroDay  and 
Minority  Scientists  Showcase,  these  professionals  demonstrated  great  enthusiasm  for  the  teens' 
learning.  Teens,  meanwhile,  could  rely  on  their  Senior  Educator  to  serve  as  a  "bridge"  to  connect  with 
the  visiting  professional.  Teens  were  more  able  to  adjust  and  grow  comfortable  with  visitors  in  the 
presence  of  a  Senior  Educator  they  trusted.  From  this,  teens  could  begin  to  see  themselves  as  future 
professionals,  stepping  into  the  shoes  of  these  special  visitors  and  achieving  a  fulfilling  career  in  STEM 
disciplines. 

Leadership  transitions  at  SLSC  factored  into  program  development  as  well.  In  the  first  months 
following  the  ONR  award,  the  President  and  CEO  of  the  SLSC  relocated  to  Seattle,  Washington,  making 
way  for  an  interim  president  through  most  of  2011,  and  the  eventual  instatement  of  current  President 
and  CEO  Bert  Vescolani.  Each  of  these  leaders  has  supported  the  efforts  of  the  YES  program.  While  no 
specific  impacts  were  obvious,  organizational  transition  associated  with  any  leadership  change  likely 
affected  some  aspects  of  program  implementation  and  momentum.  Furthermore,  in  April  2013,  YES 
founder  and  Chief  Educational  Outreach  Officer  of  SLSC  Diane  Miller  relocated  to  head  educational 
programming  at  The  Detroit  Zoo.  Siinya  Williams,  Senior  Director  of  Educational  Outreach,  assumed 
responsibility  for  the  YES  program  and  the  Community  STEM  Outreach  Program.  Williams  brought  more 
than  20  years  of  experience  at  SLSC  with  13  of  those  in  the  educational  outreach  department.  As  a 
result,  this  transition  occurred  smoothly.  As  the  program  sought  to  enhance  sustainability  beyond  ONR 
funding,  Williams  worked  with  Vescolani  and  other  leadership  at  SLSC  to  identify  ways  to  enhance 
community  support.  As  a  result,  the  program  implemented  a  new  internship-based  system  for  older 
teens  to  gain  experience  in  paid  positions  at  local  business  and  organizations.  This  system  distributes  the 
financial  requirements  of  the  YES  program  amongst  several  businesses  and  organizations  that 
participate  in  the  internship  program,  ultimately  creating  a  more  sustainable  funding  model. 

Stakeholder  Expansion 

As  described  above,  volunteer  support  played  out  more  favorably  in  advisory  or  special  guest  roles 
than  it  did  in  regular  daily  or  weekly  programming.  Three  other  elements  of  stakeholder  expansion, 
though,  proved  particularly  fruitful. 

The  program  increased  the  number  of  volunteer  educators  participating  in  program  activities. 
Beginning  in  2010  with  no  formalized  volunteer  system  and  zero  official  volunteers,  the  YES  program 
hosted  30  volunteers  at  the  end  of  2013,  having  accumulated  276  service  hours  during  the  year.  Over 
the  course  of  the  project,  the  program  averaged  44  volunteers  per  year.  In  total,  volunteers  dedicated 
more  than  1,000  hours  of  service  over  the  course  of  the  Community  STEM  Outreach  Program. 

Increasing  the  range  of  community  partners  to  include  a  more  diverse  audience  in  YES  proved 
successful.  While  the  underrepresented  population  in  St.  Louis  is  predominantly  African  American,  these 
efforts  increased  exposure  with  organizations  serving  low  income  European  Americans,  Asian 
Americans,  and  Latino(a)  Americans.  These  efforts  added  20  strong  community  partnerships  (a  50 
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percent  increase),  and  continuing  efforts  will  further  spread  awareness  of  the  YES  program.  The  YES 
program  already  maintains  a  waitlist  for  teens  who  express  interest  once  cohorts  are  at  maximum 
capacity. 

Community  partner  meetings  also  proved  very  successful.  Here,  community  partners  learned  more 
about  the  YES  program,  completed  STEM-based  activities,  and  received  curricular  materials  to  use  in 
their  own  programs.  All  community  partners  were  invited  to  attend  these  monthly  meetings,  and 
attendance  regularly  surpassed  30  participants.  Using  the  internal  SAMI  measurement  to  assess  impact, 
the  YES  Community  Partner  Meetings  achieved  an  average  score  of  14.36  (defined  as  "High  Impact") 
from  2011  to  2013. 

National  Expansion 

Evaluators  identified  early  on  that  YES  was  very  focused  on  serving  the  needs  of  its  community,  and 
the  translation  of  such  a  model  to  other  communities  would  need  to  consider  the  unique  needs  in  those 
locations.  In  response  to  this,  evaluators  created  an  exceptionally  strong  logic  model  to  guide  the 
development  of  other  programs.  This  logic  model  plays  a  significant  role  in  the  organization  of  the 
Circles  of  Support  multimedia  dissemination  tool. 

When  determining  the  potential  for  national  expansion,  evaluators  noted  that  YES  served  the 
underrepresented  population  of  St.  Louis,  but  underrepresented  populations  in  other  areas  of  the 
country  would  express  different  needs  and  come  to  the  program  with  varied  strengths.  Not  to  mention, 
each  participant  in  the  YES  program  arrives  with  individual  strengths  and  needs,  and  the  program  must 
be  flexible  enough  to  meet  the  student  where  he  or  she  enters.  Beyond  this,  other  communities  may 
have  more  or  less  trouble  with  transportation  and  safety.  They  may  or  may  not  have  access  to  leaders  in 
the  STEM  industry,  such  as  the  life  and  plant  science  professionals  available  in  St.  Louis's  Biobelt. 

Moreover,  the  degree  to  which  YES  offers  STEM  programming  may  not  be  realistic  for  a  large 
number  of  organizations.  YES,  housed  in  the  Taylor  Community  Science  Resource  Center,  has  access  to 
multiple  classrooms  spaces,  a  kitchen,  a  computer  lab,  and  a  separate  administrative  area.  SLSC  offers 
further  administrative  and  fund  raising  support  to  its  YES  program,  and  the  organization  is  committed  to 
providing  paid  incentives  to  its  teen  participants.  As  discovered  during  the  Community  STEM  National 
Collaborative  Conference  in  April  2013,  many  STEM  youth  programs  do  not  have  access  to  the  resources 
available  at  YES.  Many  indicated  that  YES  was  a  "dream  plan"  for  their  institution,  but  in  reality,  they 
were  limited  to  a  significantly  smaller  budget. 

As  a  result,  Klein  Consulting  approached  national  expansion  through  the  lens  of  program 
development  versus  program  dissemination.  Klein's  logic  model  for  the  YES  program  identified  critical 
elements  that  contributed  to  the  program's  success  in  St.  Louis.  The  dissemination  tool,  Circles  of 
Support  (www.yescirclesofsupport.com).  leads  users  through  a  program  development  process  that 
results  in  new,  revised  logic  models  specific  to  the  users'  communities.  The  process  helps  identify  the 
particular  needs  of  their  own  communities,  the  necessary  inputs,  the  program  audience,  and  ultimately 
the  outputs  and  outcomes  that  their  program  should  aim  to  obtain.  As  this  dissemination  tool  is  revised 
and  updated  by  YES  educators,  it  will  continue  to  serve  as  a  resource  for  informal  STEM  youth 
development  programs  throughout  the  country. 

Speaking  Engagements  &  Articles 

In  part  due  to  the  Community  STEM  Outreach  Program  and  strengthened  connections  between  the 
Science  Center  and  Navy  representatives,  Bert  Vescolani,  President  and  CEO  of  the  Science  Center,  was 
invited  to  speak  at  the  Navy  Ball  in  2013.  There,  Vescolani  conveyed  the  importance  of  STEM  advocates 
and  their  role  in  continued  effort  to  enhance  work  force  development  and  high-skills  training.  Diane 
Miller  and  Marlynn  Chambers  participated  in  progress  reports  at  ONR  national  meetings,  poster 
presentation  updates,  and  STEM2Stern  dialogues.  Lessons  learned  from  the  Community  STEM  Outreach 
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Program  were  also  shared  informally  through  professional  networks  such  as  the  Association  for  Science- 
Technology  Centers  (ASTC)  and  related  conferences. 

The  primary  method  of  dissemination  for  the  Community  STEM  Outreach  Program  is  the  Circles  of 
Support  multimedia  tool  (www.yescirclesofsupport.com).  The  Second  Annual  Evaluation  Report  and  the 
Summative  Evaluation  Report  are  both  available  on  www.informalscience.org.  The  project  team  did  not 
pursue  publication  of  academic  articles.  Klein  Consulting  produced  a  suggested  list  for  further  research 
(see  Appendix  C)  and  expressed  interest  in  generating  research  material  based  on  the  T0SRA2  data 
collected  during  the  ONR  project.  YES  leadership  proposed  a  number  of  additional  research  efforts 
through  programs  at  the  National  Science  Foundation  that  were  not  funded. 

In  2014  and  moving  forward,  the  Science  Center  will  play  a  role  in  Navy  Week  in  St.  Louis.  Plans  for 
2014  feature  a  diverse  three-member  panel  of  Navy  leaders.  While  plans  are  still  in  process,  tentative 
representatives  include  Vice  Admiral  William  A.  Brown  (Deputy  Commander,  U.S.  Transportation 
Command),  Rear  Admiral  Katherine  L.  Gregory  (Commander,  Naval  Facilities  Engineering  Command, 
Chief  of  Civil  Engineers),  and  Rear  Admiral  Rebecca  J.  McCormick-Boyle  (Chief  of  Staff,  Bureau  of 
Medicine  and  Surgery). 

Moving  Forward 

The  YES  program  is  appreciative  for  the  opportunities  to  try  new,  promising  program  alternatives 
over  the  past  three  years.  As  a  result  of  ONR  funding,  the  program  is  more  aware  than  ever  of  what 
drives  its  success.  YES  leaders  are  continuing  this  work  in  2014,  introducing  a  new  work-based  internship 
program  to  match  juniors  and  seniors  to  community  businesses.  This  both  enhances  the  work  skills 
development  for  teen  participants  and  reduces  the  financial  burden  on  the  SLSC  by  engaging  more 
community  support.  Moreover,  educators  continue  to  update  and  revise  STEM  content  of  curricula  to 
match  the  needs  and  interests  of  their  students.  For  example,  design  engineering  and  agriscience  will 
both  integrate  significant  new  curricular  activities  this  year,  each  addressing  timely  topics:  computer 
programming  and  sustainability.  Social  Entrepreneurship,  a  new  initiative,  will  engage  teens  in 
developing  their  own  business,  recognizing  the  strengths  and  needs  that  affect  the  success  of  startup 
business  ventures. 

The  YES  program  and  the  Saint  Louis  Science  Center  welcome  further  communication  with  the 
Office  of  Naval  Research.  Siinya  Williams,  Senior  Director  for  Educational  Outreach,  is  available 
at  srilev(5>slsc.org  or  314.289.1417,  and  Bert  Vescolani,  President  and  CEO,  is  available 
at  bvescolani@slsc.org  or  314.289.4474.  Thank  you  again  for  your  support  of  the  Community  STEM 
Outreach  Program. 
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Appendices 

A:  Circles  of  Support  Documentation 
B:  Summative  Evaluation  Report 
C:  Proposed  Research  Agenda 
D:  Proposed  Dissemination  of  YES 
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INTRODUCTION 


The  needs  of  teenagers  sometimes  get  lost  among  the  issues  of  their  struggling  families, 
schools,  and  neighborhoods.  Indeed,  their  needs  may  be  ignored  as  communities  and 
the  nation  as  a  whole  address  the  huge  task  of  developing  a  21st  century  workforce. 
More  and  more  jobs  require  science,  technology,  engineering,  and  mathematics  (STEM) 
knowledge  and  skills  to  keep  the  U.S.  economy  healthy  and  thriving  in  a  competitive 
world.  Yet,  ways  to  address  all  these  issues  may  lie  hidden,  untapped,  and  undeveloped 
within  these  very  teenagers  whose  needs  remain  low  priority.  The  highly  skilled 
workers,  good  citizens,  and  caring  parents  the  nation  needs  sit  in  classrooms  or  stand 
texting  on  street  corners,  convinced  they  have  no  future  in  STEM  education  and  careers. 
These  young  people  are  teenagers  with  multiple  risk  factors  in  their  lives.  Ignoring  their 
potential  may  be  a  risk  for  everyone.  Programs  offered  for  these  teens  during  their  out- 
of-school  time  (OST)  can  play  a  key  role  in  changing  this  situation.  The  Youth  Exploring 
Science  (YES)  Program  at  the  Saint  Louis  Science  Center  (SLSC)  is  one  such  program. 
As  Siinya  Williams,  Senior  Director  for  Community  Outreach  at  the  SLSC,  states  in  a 
video  interview  on  the  Circles  of  Support  website,  this  is  important  work:  “They  are  our 


future!” 


Foundations,  government  agencies,  and  communities  have  discovered  the  importance  of 
programs  offered  during  OST  for  children  and  youth.  Carefully  designed  programs  can 
provide  this  essential  network  of  bridges  and  highways  over  obstacles  to  STEM 
education  and  careers.  Yet  many  resources,  while  providing  excellent  information  on 
specific  approaches,  do  not  provide  a  picture  of  a  complete  working  model  designed  to 
allow  a  wide  range  of  practices  to  be  translated  into  a  coherent  system  specifically 
adapted  to  local  issues.  Diane  Miller,  founder  of  the  YES  Program  at  the  SLSC,  calls  this 
process  deliberate  design.  Deliberate  design  means  that  (1)  each  element  and  activity  in 
a  program  has  an  explicit  rationale  with  specific 
intended  outcomes;  and  (2)  each  part  of  the 
program  system  complements  the  whole. 


To  provide  a  working  model  of  such  a  system,  the 
Community  STEM  Outreach  Project  at  the  SLSC, 
with  funding  from  the  Office  of  Naval  Research 
(ONR),  undertook  the  design  of  a  multimedia  tool 
that  engages  educators  into  the  deliberate  design 
process.  This  documentation  of  the  YES  Program  is 
one  element  of  the  tool  that  provides  this  working 
model.  The  other  elements  include  short  videos, 
photographs,  program  documents,  references,  and 
design  activities.  These  can  be  found  on  the 
www.yescirclesofsupport.com  website. 


Diane  Miller,  founder  of 
the  YES  Program  at  the 
SLSC,  calls  this  process 
deliberate  design. 


Deliberate  design  means 
that  (1)  each  element  and 
activity  in  a  program  has 
an  explicit  rationale  with 
specific  intended 


outcomes;  and  (2)  each 
part  of  the  program 


system  complements  the 
whole. 
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As  part  of  the  evaluation  and  documentation  processes,  we  interviewed  YES  Program 
staff  members  who  insisted  that  all  teens  could  learn  and  succeed  if  they  were  provided 
the  support  they  needed  to  overcome  obstacles  in  their  lives.  Based  on  this  clear  vision, 
the  YES  Program,  founded  in  1998  by  Diane  Miller,  provides  circles  of  support  to  help 
these  young  people  overcome  obstacles.  Circles  of  Support  is  the  title  of  the  multimedia 
tool  of  which  this  documentation  is  a  part. 

As  a  whole,  this  multimedia  tool  provides  an  example  of  a  successful  working  model  of  a 
youth  program  for  underserved  and  at-risk  youth.  It  includes  resources  and  activities  for 
users  to  actively  engage  in  the  deliberate  design  process. 

Assumptions  underlying  the  use  of  this  tool  include  the  following: 

•  The  tool  should  be  useful  for  people  who  want  to  (1)  create  a  program;  or  (2) 
improve  an  existing  program  for  youth  ages  12  to  18  (teens)  in  OST  settings  with 
risk  factors  in  their  lives  that  provide  obstacles  to  accessing  STEM  education  and 
careers.  This  program  audience  is  sometimes  referred  to  as  underserved  youth. 
The  population  includes  substantial  numbers  of  young  people  who  are  female 
and  lower  income,  attend  schools  with  lower  than  average  rates  of  graduation 
and  college  attendance,  and  identify  themselves  as  ethnic  minorities. 

•  Particular  implementations  of  OST  STEM  youth  programs  need  to  be  adapted  to 
a  specific  community  and  specific  intuitions. 

•  Some  key  elements  of  the  YES  Program  model  are  essential  for  program  integrity 
and  to  produce  impacts  similar  to  those  of  the  YES  Program  at  the  Saint  Louis 
Science  Center. 

AUDIENCES  OF  THE  MULTIMEDIA  TOOL 

Primary  audiences  for  the  Circles  of  Support  multimedia  tool  include  leadership  and 
educational  program  staff  members  at  informal  science  learning  institutions  (i.e., 
science  museums,  natural  history  museums,  zoos,  and  gardens)  and  community 
organizations. 

Secondary  audiences  include  (1)  leaders  and  program  officers  at  foundations  and 
government  agencies  who  may  be  interested  in  funding  or  providing  technical  support 
to  institutions  with  STEM  youth  programs;  and  (2)  researchers  who  want  to  investigate 
program  elements  and  impacts  in  their  own  work. 
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LOGIC  MODEL  OF  KEY  ELEMENTS 

Organized  as  a  logic  model,  key  elements  of  the 
YES  Program  include  strategies,  tactics,  behaviors, 
and  resources  that  research  and  evaluation  findings 
indicate  are  closely  connected  with  personal, 
educational,  and  career  STEM  impacts  for  teens. 

Users  of  the  tool  should  keep  in  mind  that  this 
working  model  is  just  that,  an  ideal— how  this 
specific  program  works  at  its  best.  Like  all 
educational  efforts,  the  YES  Program  constantly 
changes  over  time,  adapting  to  local  issues  and 
events.  In  addition,  this  model  is  not  intended  for  replication.  Our  approach  focuses  on 
allowing  users  to  consider  the  needs  and  context  of  their  own  communities  and 
organizations. 

USING  THE  TOOL 

We  designed  the  tool  so  audiences  can  use  the  tool  in  a  variety  of  ways: 

•  High-level  conceptual  planning  of  program  rationale  and  structure  for  initial 
implementation  of  a  new  STEM  youth  program 

•  High-level  inventory  and  analysis  of  an  existing  STEM  program  to  plan  for 
improvement 

•  Ideas  and  frameworks  for  staff  professional  development 

•  Access  to  guidance  for  selecting  or  creating  appropriate  curriculum  for  STEM 
youth  programs 

•  Access  to  research  foundations  for  STEM  youth  programs  and  youth 
development 

The  tool  can  be  adapted  for  use  by  individuals  and  groups  as  part  of  deliberate  design 
efforts  involving  the  creation  of  a  new  program,  program  improvement,  or  as  part  of 
ongoing  professional  development  for  program  partners  or  staff  members.  In  working 
meetings  with  the  YES  staff  members,  we  asked  them  what  areas  they  believed  needed 
to  be  shared  with  people  at  other  sites  to  develop  and  implement  a  youth-based 
program.  They  identified  many  of  the  categories  that  describe  inputs  of  the  program, 
noting  that  without  the  materials,  technology,  levels  of  staff,  and  procedures,  they  could 
not  do  their  work. 

The  heart  of  the  model  is  the  Program  section.  In  this  section,  each  key  element 
description  includes  the  following  information: 

•  The  Big  Idea— an  overview  and  definition  of  the  element 

•  Description  of  Use— examples  from  the  YES  Program 

•  Deliberate  Design— the  rationale  or  reasons  for  the  use  of  the  element 

•  Results— impacts  and  outcomes  of  the  element 


This  logic  model  allows 
users  to  consider  the  needs 
and  contexts  of  their  own 
communities  and 
organizations. 
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Where  relevant,  the  description  may  also  include: 

•  Program  Documents— a  list  of  sample  documents  used  in  the  YES  Program  that 
can  be  accessed  on  the  website 

•  References— links  and  citations  to  resources  and  readings  about  the  element 
BACKGROUND 

Founded  in  1998,  the  YES  Program  has  served  more  than  600  underserved  teens 
through  work-based,  OST  programming  focusing 
on  STEM  topics  and  youth  development.  At  its 
inception,  the  YES  Program  was  part  of  the 
innovative  YouthALIVE!  Project  initiated  through 
the  Association  of  Science-Technology  Centers 
(ASTC)  in  the  1990s.  Staff  members  developed  and 
tested  research-based  practices  through  grants 
from  the  Wallace  Foundation,  the  National  Science 
Foundation,  and  other  funders,  with  goals  that 
included  providing  equity  and  access  to  STEM 
education  and  careers,  and  increasing  services  to  a 
more  diverse  range  of  community  members.  Diane 
Miller,  the  founder  and  long-time  leader  of  the  program  (1997—2013),  is  a  nationally 
recognized  leader  in  informal  STEM  learning  and  youth  development.  The  YES  Program 
is  part  of  the  Taylor  Community  Science  Resource  Center  at  the  Saint  Louis  Science 
Center,  which  received  a  2005  Roy  L.  Schafer  Leading  Edge  Award  for  providing 
leadership  in  public  outreach  and  for  its  commitment  to  increasing  participation  of 
members  from  historically  underserved  and  underrepresented  communities. 

Because  YES  was  recognized  as  a  successful  program,  the  United  States  Navy’s  Office  of 
Naval  Research  funded  efforts  to  capture  and  describe  the  program  to  share  the 
successful  aspects  the  model  and  promote  expansion  in  successful  practices  in  OST 
STEM  youth  programs. 

We  based  this  YES  Program  documentation  on  evaluation  interviews,  focus  groups, 
surveys,  document  analysis,  and  observations  conducted  by  Klein  Consulting,  the 
external  evaluator  for  the  project.  Christine  Klein  led  the  project,  and  Carey  Tisdal,  of 
Tisdal  Consulting,  was  contracted  to  assist  with  the  evaluation  and  the  focus  on  the 
design  of  the  multimedia  tool.  In  addition,  the  authors  had  access  to  photographs  and  to 
observations  conducted  by  the  YES  staff. 


Founded  in  1998,  the  YES 
Program  has  served  more 
than  600  underserved  teens 
through  work-based,  OST 
programming  focusing  on 
STEM  topics  and  youth 
development. 
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YES  LOGIC  MODEL 

A  logic  model  provides  the  organizing  framework  for  Circles  of  Support,  the  multimedia 
tool  and  this  documentation  that  is  part  of  the  tool. 

Readers  and  tool  users  may  be  familiar  with  logic 
models  through  the  work  of  the  Kellogg  Foundation 
(2004)  and  the  Institute  for  Museum  and  Library 
Services  (IMLS)-funded  online  professional 
development  offerings  through  Indiana  University- 
Purdue  University  Indianapolis  (2006).  A  logic 
model  provides  a  good  fit  to  our  task  for  several 
reasons.  Logic  models  allow  programs  and  products 
to  be  shown  visually  as  systems.  In  addition,  they 
allow  us  to  discuss  the  relationship  between 
program  elements  and  impacts;  knowing  how  a 
program  affects  the  audience  (impacts)  is 
meaningless  without  understanding  the  treatment 
or  intervention  that  influenced  the  impacts.  We 
hoped  that  logic  models  could  provide  a  consistent 
format  for  program  comparison  should  efforts  in  the 
project  be  expanded.  Finally,  logic  modeling 
provides  a  good  foundation  for  obtaining  funding 
from  businesses,  foundations,  and  government 
agencies. 


The  purpose  of  a  logic 
model  is  to  provide 
stakeholders  with  a  road 
map  describing  the 
sequence  of  related  events 
connecting  the  need  for  the 
planned  program  with  the 
program’s  desired  results. 
Mapping  a  proposed 
program  helps  you 
visualize  and  understand 
how  human  and  financial 
investments  can  contribute 
to  achieving  your  intended 
program  goals  and  can 
lead  to  program 
improvements.  (W.K  W. 
Kellogg  Foundation,  2004, 
P-3) 


Different  logic  model  frameworks  use  some  slightly  different  language,  but  all  provide  a 
way  for  people  to  see  the  logic  (or  the  theory  of  how  the  program  is  supposed  to  work)  to 
accomplish  specific  outcomes.  We  adopted  the  terms  used  in  Friedman’s  Framework 
for  Evaluating  Impacts  of  Informal  Science  Education  (2008),  which  is  widely  used  by 
informal  science  educators.  Figure  1  shows  a  hierarchy  of  anticipated  outcomes  that  is 
useful  for  developing  and  thinking  about  the  different  types  of  outcomes.  In  this 
hierarchy,  the  term  impact  refers  to  outcomes  related  to  the  changes  among  members 
of  the  target  audience  in  knowledge,  attitudes,  and  behavior.  We  chose  to  use  this  term 
not  only  for  consistency  but  also  to  make  a  clear  distinction  between  audience  impacts 
and  program  goals.  Program  goals  often  describe  activities  and  products  the  program 
staff  intends  to  accomplish.  In  contrast,  impacts  are  the  desired  changes  in  the 
knowledge,  attitudes,  and  behaviors  among  audience  members. 
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Hierarchy  of  Anticipated  Outcomes 


Strategic  impacts- \ 
individual. 

Community,  Environ¬ 
mental,  Economic 


Long  Term  -  retention  and 
lasting  changes  in 
knowledge,  attitude,  and/or 
behaviors 


Short  Term  -  comprehension  and  initial  changes  in 
knowledge,  attitude,  and/or  behaviors 


People  -  number  of  people/ visitors  affected 
Programs  -  number  of  programs  provided 
Products  -  number  of  products  distributed 


Outcomes/!  mpacts 


Outputs 


Psychographic  data  -  motivations,  interests,  existing  knowledge,  expectations, 
perceptions,  etc. 

Descriptive  data  -  ages,  ethnicity,  gender,  income,  residence,  distance  traveled,  group  size, 
etc. 


Psychographics/ 

Demographics 


Figure  1.  Hierarchy  of  Anticipated  Outcomes  Adapted  from  Friedman  (2008,  p.  26) 

Like  any  tool,  logic  models  have  their  limitations.  First,  it  is  important  to  note  that  the 
program  does  not  cause  the  impacts— it  influences  them.  The  YES  Program  is  a  complex 
system,  embedded  in  other  complex  systems  such  as  the  community,  school  systems, 
and  the  national  economy  and  culture. 

The  YES  Logic  Model  has  the  following  elements: 

•  Inputs  include  the  foundational  assumptions,  needs,  and  resources  supporting 
the  program  as  well  as  the  partnerships  and  professional  development  required 
for  the  program  to  operate.  Some  logic  model  processes  break  these  into  different 
groups:  conceptually  and  visually.  In  this  documentation,  they  are  grouped  to 
indicate  they  are  basic  elements  that  are  needed  for  the  YES  Program  to  operate 
with  teens. 

•  Audience  includes  the  characteristics  of  the  target  audience  of  the  program. 

•  Program  describes  the  activities  through  which  the  audience  outcomes 
(impacts)  are  accomplished. 

•  Outputs  include  the  number  of  people  served  and  the  number  and  types  of 
program  offerings. 

•  Impacts  describe  the  short-  and  long-term  changes  in  knowledge,  attitudes,  and 
behavior  among  members  of  the  target  audience. 

•  Evaluation  describes  the  ongoing  processes  of  asking  and  answering  questions 
about  the  value  of  the  program  and  its  component  parts. 
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Figure  2  presents  the  YES  Program  Logic  Model.  The  standard  elements  of  a  Theory  of 
Change  Logic  Model  run  across  the  top  of  the  model.  Elements  in  this  model  are  used  to 
structure  this  documentation  and  organize  the  Circles  of  Support  website. 


Audience 

Program 

Outputs  ] 

"  Impacts 

Need  Learning  Strategies  H  Attendance  H  Short-term 

^  Program  Structure  J  Learning  Labs  ^  Long-term 

Community  Involvement  B  J  New  Teens  H  Partners  s  ?wj4  X7  < 

Professional  Development  STEM  Components  H  Contact  Info 

Program  Resources 

|  College  Pi  ep 

Evaluation 


Figure  2:  YES  Program  Theory  of  Change  Logic  Model 
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INPUTS 


Inputs,  in  the  YES  Logic  Model,  include  Needs, 

Program  Foundations,  Community  Involvement, 

Professional  Development,  and  Program  Resources. 

Needs,  as  recognized  and  understood  by  the 
program  stakeholders,  serve  as  an  important  input 
into  the  program  system.  Stakeholders  are  groups  of 
people  who  can  benefit  from  the  program  or  who 
have  something  at  risk  based  on  whether  or  not  the 
program  exists  and  how  it  operates.  For  the  YES 
Program,  important  stakeholders  include  teens  (the 
program  audience),  teens’  parents,  staff  members  in 
the  YES  Program,  SLSC  administrators,  the  St.  Louis  business  community,  community- 
organizations,  local  schools,  and  funders  of  grant -based  projects  that  are  part  of  the 
program.  Through  evaluation  interviews,  we  found  that  different  stakeholding  groups 
had  somewhat  different  understandings  of  the  needs  the  YES  Program  serves  but 
generally  can  agree  about  the  importance  of  most  of  the  impacts. 

Program  Foundations  describes  the  important  concepts  that  underpin  the  YES 
Program.  These  concepts  reflect  the  basic  assumptions  program  staff  members  assume 
to  be  true  and  reflect  their  philosophy  of  education. 

Community  Involvement  serves  as  a  defining  aspect  of  the  YES  Program.  Through 
community-based  organizations  and  other  partners,  the  YES  Program  joins  with  and 
strengthens  the  network  of  support  for  teen  participants. 

Professional  Development  defines  the  processes  through  which  staff  members  from 
diverse  backgrounds  (e.g.,  formal  education,  informal  education,  social  work,  scientific 
research,  and  engineering)  build  shared  understandings,  knowledge,  and  skills  to  offer 
the  program  in  consistently  effective  ways  and  make  ongoing  improvements. 

Program  Resources  include  the  people,  materials,  and  systems  necessary  for  the 
program  to  work.  Program  scope  and  design  depend  on  certain  levels  of  resources  for 
consistent  and  effective  operation  and  ongoing  improvement. 


Inputs  include  the 
foundational  assumptions, 
needs,  and  resources 
supporting  the  program  as 
well  as  the  partnerships  and 
professional  development 
required  for  the  program  to 
operate  effectively. 
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NEEDS 


Program  design  and  function  should  meet  clear 
and  important  needs  recognized  by  and  relevant  to 
the  community  and  institution  of  which  it  is  part 
and  the  individuals  it  serves.  A  need  is  a  gap 
between  the  current  situation  and  the  desired 
situation.  This  gap  can  also  be  considered  a 
problem  the  program  solves.  If  funding  from 
foundations  and  government  agencies  is  required, 
the  program  must  also  meet  needs  at  a  national  or  international  level.  Different 
stakeholding  groups  may  value  some  needs  more  highly  than  others.  Empirical  evidence 
obtained  through  existing  or  site-specific  research  and  evaluation  clarifies  and  verifies 
needs. 

In  documenting  the  YES  Program,  we  found  three  different  versions  of  need  held  by 
different  program  stakeholders.  Business  leaders  and  government  agencies  cited  a  lack 
of  individuals  prepared  for  STEM  careers  as  well  as  workers  needed  to  support  a  healthy 
economy  and  secure  armed  forces.  Leaders  of  the  Saint  Louis  Science  Center,  along  with 
national  organizations  representing  other  museums  and  science  centers,  reported  the 
historic  underrepresentation  of  low-income  citizens  and  people  of  color  among 
attendees  at  science  museums  and  museums  in  general.  They  saw  youth  programs  as  a 
way  to  build  bridges  and  provide  fair  and  relevant  services  to  all  segments  of  the 
community.  Finally,  among  YES  Program  staff  members,  Community  Partners,  and 
leaders  at  some  national  organizations,  the  needs  were  stated  as  a  lack  of  equity  in 
access  to  STEM  education  and  careers.  Succinctly,  these  versions  of  need  can  be  labeled 
as  workforce  development,  fair  service,  and  access  and  equity.  The  design  of  the  YES 
Program  at  the  Saint  Louis  Science  Center  addresses  each  of  these  needs. 

The  following  snapshots  provide  a  summary  of  how  these  needs  were  stated: 

Workforce  Development:  The  United  States  as  a  country  and  St.  Louis  as  a 
city  need  more  people  in  the  STEM  workforce  for  a  healthy  economy.  Some 
groups  (e.g.,  females,  people  of  color,  and  youth  from  lower -income  families)  are 
underrepresented  in  the  STEM  workforce;  by  plugging  the  leaky  pipeline  to 
STEM  careers  for  these  groups,  the  labor  force  will  be  stronger  and  the  economy 
healthier.  Science  museums  can  work  together  with  businesses  and  community 
groups  to  plug  this  leaky  pipeline. 

Fair  Service:  Institutions,  especially  those  such  as  the  Saint  Louis  Science 
Center  that  are  supported  by  taxpayers,  have  the  responsibility  to  meet  the 
needs  of  the  entire  community  through  their  missions.  There  is  a  gap  between 
the  demographics  of  through-the-door  visitors  and  those  of  the  St.  Louis 
community,  indicating  lower  service  to  lower-income  and  non-white  citizens. 
Science  museums  meet  the  need  of  higher-income,  white  citizens  through  leisure 
time  visit  experiences  (galleries,  events,  and  theaters).  To  meet  the  needs  of 
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ethnic  groups  without  this  cultural  tradition,  outreach  efforts  are  needed  to  build 
relationships  and  provide  fair  levels  of  service  throughout  the  entire  community. 

Access  and  Equity:  Many  promising  young  people  from  low-income  families, 
people  of  color,  and  girls  don’t  have  access  to  STEM  education  and  careers  due 
to  inequitable  life  experiences  and  social  obstacles.  Most  formal  education 
structures  assume  equal  life  experiences— an  assumption  that  reinforces  social 
obstacles.  By  providing  experiences  to  remove  these  obstacles,  science  museums 
and  community  organizations  are  well  placed  to  fill  this  gap.  Examples  of  the 
results  of  these  barriers  include  the  following: 

•  Young  people  in  these  under  served  groups  may  be  tracked  into  lower  level 
courses  in  the  schools  they  attend.  Thus,  they  may  not  take  the  appropriate 
science  and  math  courses  to  enter  as  college  STEM  majors. 

•  Young  people  may  not  see  themselves  as  someone  in  a  STEM  career  due  to 
the  lack  of  role  models  among  family,  friends,  and  acquaintances. 

•  Some  young  people  do  not  have  the  foundational  experiences  that  support 
STEM  learning.  A  good  example  of  this  is  the  water  play  in  which  middle- 
class  children  engage  at  home  or  at  preschool.  Without  the  opportunity  to 
“make  waves,”  young  people  may  lack  the  underlying  experience  for  wave 
forms  in  physics  and  earth  science. 

Response  to  the  Workforce  Development  need  is  reflected  in  several  of  the  YES  Program 
Learning  Strategies  described  later  in  this  document.  These  include  the  Work-based 
Program  and  the  Career  and  College  Readiness— Developing  21st  Century  Skills 
Learning  Strategies.  In  addition,  STEM  Components  build  in-depth  STEM  knowledge 
and  skills.  Response  to  the  Fair  Service  need  is  reflected  in  the  high  level  of  community 
involvement  in  the  YES  Program.  Finally,  the  Access  and  Equity  perception  of  need 
underpins  the  instructional  design  of  the  program. 
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PROGRAM  FOUNDATIONS 

Based  on  evaluation  interviews  and  document 
analysis,  evaluators  identified  five  foundational 
elements: 

•  Research-based  Practice 

•  Deliberate  Design 

•  Relationships 

•  Learning  in  Context 

•  Prolonged  Engagement 

These  foundational  concepts  underlie  the  decision 
making,  design,  and  practice  throughout  the  YES  Program. 


Foundational  concepts 
underlie  the  decision  - 
making,  design,  and 
practice  throughout  the  YES 
Program. 


RESEARCH-BASED  PRACTICE 

When  possible,  YES  Program  design  is  based  on  solid  research  from  a  variety  of  fields 
including  youth  development,  work- place  skills  development,  education,  learning 
sciences,  STEM  education,  and  informal  learning  environments.  Most  of  the  Learning 
Strategies  in  the  YES  Program  are  based  on  research.  Prime  examples  of  this  approach 
are  the  research-based  practices  reviewed  in  Taking  Science  to  School  (NRC,  2007)  and 
Learning  Science  in  Informal  Environments  (NRC,  2009). 

Research-based  practice  provides  logical  frameworks  (theories)  and  evidence  to  make 
decisions  about  what  works  and  what  does  not  in  specific  situations.  In  addition,  staying 
up-to-date  on  research  findings  provides  a  constant  source  for  program  improvement 
and  innovation. 


DELIBERATE  DESIGN 

Each  element  and  practice  in  the  YES  Program  has  a  specific  rationale  and  purpose  in 
the  overall  system.  Each  element  of  the  program  has  specific  intents  and  is  used 
knowingly  with  a  specific  outcome  in  mind.  Deliberate  design  shares  some  similarities 
to  formal  instructional  design  models,  but  managers  and  staff  use  this  idea  more 
broadly  to  mean  intentional  decision  making  informed  by  theory  and  evidence. 

Each  semester,  managers  and  educators  make  decisions  about  the  specific  focus  of 
STEM  Components,  but  the  framework  for  making  the  decisions  is  based  on  a 
consistent  rationale.  The  range  of  STEM  Components  includes  concepts  that  can 
roughly  be  categorized  as  life  science,  physical  science,  chemistry,  and  engineering  to 
allow  teens  with  interests  and  enthusiasm  for  specific  areas  to  pursue  their  interest  over 
a  four-year  timeframe.  In  addition,  each  component  must  be  focused  around  an 
important  community  need  so  STEM  concept  learning  occurs  in  context.  For  example, 
for  teens  that  live  in  food  deserts,  the  Agriscience  Component  focuses  on  access  to 
healthy  food. 
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Each  semester,  educators  use  a  Curriculum  Planning  Template  that  documents  a 
component  description  in  terms  of  goals,  objectives,  and  a  schedule  of  activities  as  well 
as  making  explicit  connections  to  the  two  overarching  Learning  Strategies,  the  7 
Developmental  Needs  of  Youth  (YMCA  of  the  USA,  1998),  and  the  21st  Century  Skills 
(Stuart  &  Dahm,  1999). 

As  in  any  youth  program,  sometimes  teens  behave  inappropriately.  Many  YES  Teens 
come  from  schools  with  zero-tolerance  policies  related  to  inappropriate  behavior  where 
the  behavior  is  identified  and  the  teen  is  punished.  In  the  YES  Program,  when  teens 
behave  inappropriately  (losing  their  temper,  not  wearing  their  work  uniform, 
consistently  showing  up  late),  managers  and  staff  members  make  decisions  influenced 
by  two  Learning  Strategies.  The  7  Developmental  Needs  of  Youth  contributes  the  idea 
that  adolescents  learn  and  develop  more  successfully  with  clear  limits.  This  means  that 
the  inappropriate  behavior  needs  to  be  addressed.  Another  strategy,  providing  a 
Welcoming  and  Safe  Environment  for  Identity  Development,  contributes  the  idea  that 
the  YES  Program  should  allow  students  to  leave  behind  ideas  about  themselves  as 
someone  who  is  not  good  at  science  or  someone  who  is  a  behavior  problem.  Staff 
members  guide  teens  to  move  past  labeling  the  behavior  as  inappropriate  and  to  move 
on  to  identifying  the  feelings  and  situations  underlying  the  behavior.  Then,  staff 
members  help  teens  can  find  methods  to  rectify  the  problem  situation.  For  many  teens 
in  the  YES  Program,  this  application  of  Deliberate  Design  provides  the  scaffolding  to 
help  them  become  adult  problem-solvers  who  see  themselves  as  capable  and  resilient. 

Using  Deliberate  Design  as  a  process  to  shape  decisions  creates  a  context  where  staff 
members  are  expected  to  base  their  decisions  on  clear  rationale  and  shared 
assumptions.  These  assumptions  are  tested  in  how  efficiently  the  program  works. 
During  the  evaluation  of  the  Community  STEM  Outreach  Program,  managers  and 
educators  both  expressed  dissatisfaction  with  how  behavior  problems  were  handled. 
After  considerable  discussion,  an  explicit  framework  was  developed,  one  that 
represented  both  the  teens’  need  for  clear  limits  and  need  for  a  safe  environment  to 
develop  new  and  more  positive  identities. 


LEARNING  IN  CONTEXT 

Learning  in  the  YES  Program  is  situated  within  meaningful  work,  a  project,  or  inquiry- 
based  problem  solving.  This  approach  to  learning  contrasts  to  some  formal  education 
environments  where  concepts  are  presented  in  isolation  from  how  they  are  applied. 

In  the  YES  Program,  examples  of  meaningful  work  include  teaching  younger  children 
from  a  community-based  organization  or  homeless  shelter,  growing  carrots  for  a  local 
food  pantry,  and  writing  a  resume  to  apply  for  a  YES  Summer  Internship.  Projects  teens 
undertake  in  the  YES  Program  vary  considerably  in  length.  Brief,  one-session  activities 
include  designing  rockets,  making  healthy  snacks  from  a  recipe,  and  determining  the 
temperature  at  which  specific  biofuels  burn.  Some  project -based  learning  spans  entire 
semesters,  including  building  robots  for  competition,  designing  and  building  an 
aquaponic  system  to  grow  food,  or  designing  and  prototyping  exhibits  for  Science 
Center  visitors.  Inquiry-based  problem  solving  begins  with  simple  inquiry  activities  such 
as  exploring  the  change  in  temperature  when  salt  is  added  to  ice  in  the  process  of 


Klein  Consulting— Version  1.0— October  15, 2013 


12 


A17 


making  ice  cream  and  extends  to  longer-term  inquiry  projects  such  as  developing 
hypotheses  about  the  effect  of  light  levels  on  plant  growth  and  testing  the  hypothesis 
through  systematic  experiments. 

Learning  in  context  makes  learning  essential  to  performing  that  work,  completing  a 
product,  or  solving  a  problem;  that  is,  the  context  makes  the  learning  immediately 
relevant.  In  addition,  some  teens  enter  the  YES  Program  without  the  experience  with 
materials  and  phenomena  that  is  assumed  by  writers  of  textbooks  and  required  to  deal 
with  concepts  abstractly.  The  YES  Program  provides  these  types  of  experience. 


RELATIONSHIPS 

Interpersonal  and  group  relationships  support  the  communication,  commitment,  and 
flexibility  that  allow  the  YES  Program  to  function  as  a  coherent  system.  Relationships 
work  at  several  levels  of  the  program. 

Key  relationships  within  the  program  include  those  between  staff  members  and  teens, 
and  relationships  among  groups  of  teens  working  together.  Educators  serve  as  mentors 
and  role  models,  building  relationships  with  the  teens  as  they  demonstrate  appropriate 
workplace  behavior,  describe  their  college  experiences,  comment  on  journal  entries  and 
blogs,  and  set  the  tone  for  STEM  learning.  Teens  build  relationships  with  one  another, 
telling  evaluators  that  they  make  new  friends  in  YES  with  some  people  so  different  from 
themselves  they  would  not  have  previously  considered  as  friends. 

As  one  summer  intern,  a  college  student,  said: 

The  most  valuable  aspects  that  I  see  of  the  YES  Program  is  being  able  to  build  a 
relationship  with  the  teens,  having  one-on-one  relationships  with  them,  because 
then  it’s  not  so  hard  to  get  them  interested,  and  you  get  to  know  them  so  you  get 
to  present  more  things  that  they  find  interesting.  (August  2012  YES  Summer 
Intern  Focus  Group) 

Relationships  among  Community  Partners  and  the  SLSC  and  YES  Program  staff 
members  provide  another  key  element.  Community  Partners  recruit  teens  for  the 
program  and  bring  younger  children  to  summer  programs  led  by  the  YES  Teens.  A 
youth  program  such  as  YES  cannot  do  everything.  Some  Community  Partners  offer 
social  and  psychological  services  needed  by  some  teens;  others  offer  locations  for 
internships;  and  still  others  provide  volunteers  for  Learning  Labs  and  for  presentations 
at  events. 


PROLONGED  ENGAGEMENT 

Ongoing,  year-round  engagement  in  the  YES  Program  allows  for  in-depth  learning  well 
beyond  that  accomplished  in  a  typical  workshop  or  summer  camp  experience.  Prolonged 
engagement  also  supports  the  development  of  sustained,  supportive  relationships 
between  teens  and  program  staff  and  among  the  teens  themselves  that  allows  the 
program  to  respond  to  individual  needs,  personalities,  and  goals. 
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Several  strategies  support  prolonged  engagement: 

•  Saturday  sessions  offered  during  the  school  year  keep  teens  involved  in  the 
program  year-round. 

•  Educators  call  and  text  teens  if  they  do  not  attend  for  several  sessions  to  find  out 
how  they  are  and  if  there  are  any  solvable  problems. 

•  Teens  may  drop  out  a  semester  or  two  to  participate  in  sports  or  attend  to  family 
issues  and  then  be  welcomed  back  to  enroll  in  subsequent  semesters. 
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COMMUNITY  INVOLVEMENT 


|  THE  BIG  IDEA 

Some  youth  programs  partner  primarily  with  schools,  but  in  the  YES  Program,  the 
primary  partners  are  community-based  organizations.  The  YES  Program  also  partners 
with  schools,  parents,  scientists,  and  science-based  businesses. 

|  DESCRIPTION  OF  USE 

As  of  2013,  the  YES  Program  had  developed  about  65  partnerships.  About  35  of  these 
were  Community  Partners,  and  30  were  Resource  Partners.  Community  Partners  were 
primarily  community-based  organizations  that  serve  teens  and  their  families,  typically 
urban,  low-income  populations.  A  few  were  schools  that  serve  specific  populations;  for 
example,  the  International  Studies  Program  of  St.  Louis  Public  Schools  that  serves 
immigrants  to  the  city— St.  Louis  has  the  largest  Bosnian  population  outside  of  Bosnia 
and  increasing  numbers  of  Spanish  speakers  and  Asian  immigrants.  Resource  Partners 
were  those  who  work  directly  with  teens— coaching  the  First  Robotics  team,  presenting 
their  research  in  the  neuroscience  components,  or  providing  STEM  and  business 
guidance  such  as  HOSCO  Farms,  which  focuses  on  local  food  production. 

Monthly  Community  Partner  meetings  offer  an  opportunity  for  Community  and 
Resource  Partners  from  various  organizations  to  engage  in  professional  development 
activities  led  by  YES  educators  and  to  network  with  one  another  over  lunch.  This 
monthly  meeting  supports  the  partners,  which  in  turn  supports  their  ability  to 
understand  and  support  the  YES  Program  and  the  YES  Teens. 

Community  Partners  recruit  teens  for  the  YES  Program.  Because  demand  for  admission 
to  the  program  is  high,  YES  staff  members  send  Community  Partners  specific  numbers 
of  applications.  Each  application  is  numbered  to  ensure  that  applications  aren’t 
duplicated.  For  example,  the  staff  sends  five  applications  to  the  Sea  Cadet  Program  at 
the  Cleveland  Naval  Junior  Academy.  Boys  and  Girls  Club  of  America  gets  three 
applications.  Annie  Malone  Children’s  and  Family  Service  Center  gets  three 
applications. 

Partnerships  are  a  two-way  street.  Summertime  Science,  a  YES  Component,  supports 
Community  Partners  by  offering  science  and  math  experiences  to  children  in  programs 
offered  by  partners.  Children  from  these  organizations  come  to  the  Science  Center 
during  the  summer  to  participate  in  programs  led  by  YES  Teens.  On  their  end, 
Community  Partners  provide  a  network  of  social  services  that  may  be  needed  by  YES 
Teens  and  offer  sites  for  Internships. 

People  from  some  resource  partners  work  directly  with  teens.  Engineers  coach  teens  in 
the  robotics  components.  A  counselor  from  St.  Louis  Community  College  at  Florissant 
Valley  did  a  resume  writing  workshop  as  part  of  college  prep.  Some  partners  make  a 
major  commitments,  for  example,  the  CEO  from  HOSCO  Farms  personally  led  a  2013 
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summer  Internship  Component  for  16  YES  Teens  and  provided  materials  and  expertise 
supporting  the  Agriscience  STEM  Component. 

Events  provide  valuable  connections  for  partners  and  YES  Teens.  Partners  all  come 
together  in  events  such  as  the  Minority  Scientist  Showcase.  Flyers  and  letters  publicizing 
this  event  go  out  to  inform  school  principals  and  counselors,  and  all  Community 
Partners.  Minority  scientists  from  all  over  the  St.  Louis  area  present  demonstrations  and 
talk  with  parents  and  young  people.  At  the  23rd  annual  Minority  Scientist  Showcase  in 
2013,  more  than  15,000  people  attended!  Events  like  this  make  the  Saint  Louis  Science 
Center  a  valuable  resource  for  families  and  youth  in  the  community  and  send  the 
message  that  STEM  education  and  STEM  careers  are  possible  and  open  to  everyone. 

At  networking  events,  volunteers  join  YES  Teens  and  staff  members  at  tables  for  small 
group  discussions  about  careers  in  business,  the  arts,  and  STEM  occupations.  YES 
Teens  practice  communications  skills  and  learn  about  careers  and  jobs  in  their  own 
community. 


|  DELIBERATE  DESIGN 

The  Deliberate  Design  for  community  involvement  includes  the  following  rationale: 

•  Focusing  on  two-way  partnerships  with  community-based  organizations  makes 
the  SLSC  a  key  network  player  in  community  improvement  efforts  in  the  St.  Louis 
area  and  works  toward  a  goal  of  fair  service  to  all  members  of  the  community. 

•  Recruitment  through  Community  Partners,  rather  than  nominations  from 
teachers  and  school  counselors  as  in  some  other  OST  programs,  allows  teens  to 
enter  the  YES  Program  and  develop  new  identities. 

•  Community  Partners  know  teens  and  know  the  YES  Program;  they  are  best 
equipped  to  make  judgments  about  who  will  benefit.  This  prevents  the  program 
from  becoming  focused  only  on  teens  already  doing  well  in  school. 

•  Youth  programs  cannot  provide  all  the  types  of  services  needed  by  teens  from 
low-income,  urban  neighborhoods.  Community-based  organizations  provide 
other  support  services  and  resources  teens  and  the  program  need. 


|  RESULTS 

In  a  focus  group  with  Community  Partners  in  May  2012,  partners  described  their 
involvement  with  and  appreciation  for  the  YES  Program.  Networking  among  partners 
has  benefits  beyond  the  boundaries  of  the  program,  benefits  that  reach  teens  and  youth 
in  many  ways.  In  the  first  quote,  an  example  of  the  rich  networking  among  the 
community,  one  partner  describes  how  another  partner  connected  him  with  the  YES 
Program. 

Our  organization  assists  children  who  have  a  parent  [who’s]  incarcerated.  And 
the  children  are  inner  city  underprivileged  children  from  the  ages  of  5  to  17.  So 
sometimes  they  don’t  get  exposed  to  things  like  that  [STEM]  as  often  as  other 
children  who  go,  you  know,  to  other  schools  and  things  like  that  would  get.  So 
I’m  just  saying  that  what  you  said  about  the  networking  and  sharing  resources, 
stuff  like  that  [partner]  over  here,  who  was  actually  on  our  board,  she 
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introduced  me  to  Siinya  who  was  part  of  the  YES  Program.  That’s  how  our 
organization  came  to  the  teaching  of  the  YES  Program....  Then  we  started 
working  with  the  Science  Center,  coming  out,  seeing  what  they  were  doing,  and 
programs  that  they  were  doing  and  stuff  like  that. 

In  addition,  Monthly  Community  Partner  meetings  extend  STEM  learning  experiences 
and  teaching  expertise  beyond  YES  Program  boundaries. 

One  of  my  colleagues,  this  was  her  thing  that  she  was  doing  once  a  month 
coming  to  the  community  thing  [Community  Partners  monthly  meeting].  So  my 
boss  asked  me  if  I  could  come  so  I’d  have  experience,  since  I  am  in  charge  of  the 
science  component  for  our  summer  program.  So  I figured  that  I  might  be 
coming  to  the  different  sessions  each  month,  and  I  could  learn  about  different 
types  of  science  topics  and  components. 

Another  Community  Partner  described  the  many  ways  some  her  organization  had 
interacted  with  the  Saint  Louis  Science  Center  to  strengthen  the  sense  of  community 
and  support  the  program. 

We’ve  had  YES  Teens  come  and  work  with  our  kids  during  the  summer.  We  also 
come  to  the  Summertime  Science.  We  have  kids  we’ve  submitted  for  YES.  So 
actually  that’s  very  good  for  our  kids,  because  they  see  the  older  kids  that  can 
get  a  job  at  the  Science  Center  and  sometimes  we  come  over  and  they  go,  oh, 
that’s  where  such-and-such  works,  we’re  going  there  today?  So  they  really  like 
that.  Last  week  or  maybe  two  weeks  ago,  we  had  a  career  fair  in  this  building. 
That’s  one  of  the  perks  of  being  a  Community  Partner  that  you  can  use  the 
building.  So  we  had  a  career  fair,  and  we  served  the  community.  We’ve  also  had 
our  kids  come  for  special  events  here.  One  time  they  had  a  music,  PULSE,  where 
they  did  a  lot  of  dancing  and  taught  the  kids  about  music.  It  was  one  Saturday, 
so  we  had  the  Saturday  field  trip. 


\  PROGRAM  DOCUMENTS 


YES  Application 

YES  Community  Partnership  Agreement 

YES  Community  Partner  Activity— Bingo  and  Wildlife  Rescue 

YES  Activity  Community  Partner  Activity— Microwaveable  Plastic 

YES  Activity  Community  Partner  Meeting— Toothbrush  Tackle  &  Brainstorming 
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PROFESSIONAL  DEVELOPMENT 


I  THE  BIG  IDEA 

Operation  of  the  YES  Program,  as  well  as  program  design  and  ongoing  improvement, 
depends  on  managers  and  educators  who  have  the  understanding,  skill  sets,  and 
attitudes  to  work  effectively  as  a  team  to  build  relationships  with  teens  and  partners, 
plan  curriculum  based  on  specific  learning  strategies,  and  comfortably  handle  both 
youth  development  and  STEM  learning  responsibilities. 


|  DESCRIPTION  OF  USE 

As  part  of  the  evaluation  supporting  the  documentation,  we  identified  several  different 
types  of  Professional  Development  (PD)  offered  for  YES  staff  members.  Some  PD  was 
led  by  staff  members  themselves  to  develop  their  skills  in  inquiry-based  learning  and 
facilitate  design-based  projects.  Other  PD  was  led  by  outside  experts. 

Program  and  staff  participation  in  grant -based  and  nationally  recognized  PD  efforts 
provided  opportunities  for  some  PD  sessions.  For  example,  the  SLSC’s  ongoing 
collaboration  with  the  Educational  Development  Center’s  National  Partnership  for 
Afterschool  Science  (NPASS)  led  to  PD  opportunities  focusing  on  inquiry  and  design 
activities  developed  in  these  grant-based  projects.  In  addition,  some  YES  staff  members 
participated  in  national  programs  such  as  those  offered  by  the  Institute  for  Inquiry  at 
the  Exploratorium  .  After  such  opportunities,  staff  returned  to  organize  and  lead 
training  sessions  for  other  YES  staff  members.  A  grant  project  based  on  the  Teach  to 
Learn— Learn  to  Teach  strategy  led  to  PD  in  Reflective  Practice,  a  strategy  needed  to 
implement  the  grant-funded  program  developed  collaboratively  by  five  different 
institutions. 

YES  Program  leadership  also  networked  with  nationally  recognized  experts  in  equity 
and  diversity,  such  as  Eric  Jolly,  President  and  CEO  of  the  Science  Museum  of 
Minnesota.  Dr.  Jolly  led  sessions  with  both  staff  members  and  teens,  sharing  his 
strategies  and  expertise. 

Evaluators  also  found  PD  offered  through  supportive  relationships  among  and  between 
staff  members.  Some  staff  members  enter  the  program  with  backgrounds  in  social  work 
and  youth  development.  As  one  YES  staff  member,  fresh  from  working  at  a  neuroscience 
laboratory  and  bringing  her  own  considerable  knowledge  and  expertise  to  the  program, 
commented  about  a  colleague,  “She  was  my  youth  development  mentor.  I  have  learned 
so  much  from  her  about  how  to  do  this  work.” 

Staff  members  entering  the  program  with  STEM  backgrounds  supported  the  informal 
PD  for  their  colleagues.  An  educator  with  a  degree  in  nuclear  engineering  collaborated 
with  other  educators  in  Agriscience  and  Exhibit  Lab  STEM  Components  by  identifying 
STEM  concepts  and  sharing  his  expertise.  Finally,  most  new  staff  member  orientation 
took  place  by  having  more  experienced  staff  share  information  and  examples,  and 
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answer  questions.  While  this  process  is  valuable,  some  formal  elements  of  the  program 
were  challenging  for  new  staff  members  to  incorporate  quickly  into  the  repertoire. 


|  DELIBERATE  DESIGN 

PD  for  YES  Program  staff  members  underpins  the  implementation  of  the  program  as 
designed  to  accomplish  the  intended  impact.  Without  consistent  comprehensive  PD  the 
implementation  of  the  program  gets  off  track,  with  some  learning  strategies 
misunderstood  and  misapplied. 


|  RESULTS 

When  new  staff  members,  who  were  hired  in  January  2011  through  ONR  funding, 
participated  in  PD,  they  were  oriented  to  many  of  the  learning  strategies.  As  they 
worked  with  existing  YES  educators,  these  new  staff  members  further  developed  their 
understanding.  When  educators  participated  in  workshops  led  by  a  Senior  Educator 
trained  through  NPASS,  they  further  developed  their  ability  to  lead  inquiry-based 
activities  with  teens  in  their  components.  When  educators  participated  in  workshops  led 
by  experts  like  Dr.  Jolly,  they  increased  their  understanding  of  the  needs  of  the  teens. 

At  the  same  time,  as  indicated  by  findings  from  the  Community  STEM  Outreach 
project’s  evaluation,  when  no  strategy  to  orient  summer  part-time  staff  to  the  program 
existed,  it  led  to  inconsistent  use  of  strategies,  such  as  Reflective  Practice  and  Teach  to 
Learn  -  Learn  to  Teach.  When  workshops  for  educators  did  not  include  opportunities  to 
immediately  apply  the  strategies  covered  in  the  workshop  and  did  not  receive  follow-up 
support,  they  found  the  workshops  less  than  helpful. 


I  PROGRAM  DOCUMENTS 


YES  Klein  Reflective  Practice  PD 

YES  Orientation  Schedule  Summer  2012 

Youth  Exploring  Science  Summer  Intern  Manual  2012 
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RESOURCES 

Out-of-school  time  (OST)  youth  programs  require  adequate,  ongoing  resources  to 
accomplish  their  goals.  Specific  types  of  resources  are  required  to  operate  a  program  for 
a  specific  target  audience  with  specific  learning  strategies  and  intended  impacts.  As  part 
of  the  evaluation  that  supported  the  development  of  the  documentation  of  the  YES 
Program,  we  identified  six  types  of  resources  required  by  the  YES  Program. 

•  Materials 

•  Technology 

•  Space 

•  Staff 

•  Budget 

•  Ongoing  Funding 


MATERIALS 

As  part  of  curriculum  planning  for  learning  labs,  educators  analyze  and  submit  detailed 
lists  of  materials  that  will  be  required  that  semester.  These  items  are  listed  on  Purchase 
Orders,  included  as  part  of  the  Curriculum  Planning  Template  so  the  required  materials 
are  provided  to  the  educators  by  the  point  in  the  semester  when  they  will  be  used. 

The  YES  Program  requires  general  resources  for  office  and  Learning  Lab  use.  These 
include  such  things  as  journals,  flip  charts,  markers,  poster  board,  Post-its,  and  pens. 

In  addition,  STEM  Components  require  specialized  resources.  For  example: 

•  Engineering  and  Design  Science  Components  require  materials  such  as  robotics 
kits,  building  materials,  and  safety  goggles. 

•  Energy  and  Biofuels  uses  fuel  sources  that  must  be  purchased,  as  well  as  empty, 
used  aluminum  cans  that  need  to  be  collected  and  stored. 

•  Agriscience  materials  include  soil,  seeds,  plants,  and  grow  lights,  as  well  as  water 
pumps  and  tubing  for  the  construction  of  aquaponic  systems.  Recycled  plastic 
bottles  are  also  required  for  aeroponic  systems.  For  cooking  classes,  food  items 
are  needed. 

•  Mystery  of  Matter  requires  chemicals,  glassware,  and  safety  equipment. 

•  Components  such  as  Neuroscience  and  Agriscience  sometimes  involve  teens 
caring  for  and  studying  animals. 

The  purchasing,  storing,  and  managing  of  many  materials  are  assigned  to  two  support 
staff  members.  After  processing  purchase  orders,  they  place  online  orders  where 
possible  but  also  travel  to  hardware  stores  and  retail  stores  to  purchase  and  transport 
items  such  as  lumber,  food,  and  small  equipment.  These  staff  members  managed  a 
clearly  labeled  storage  area,  keeping  track  of  inventory  until  it  is  needed,  providing 
items  to  educators  in  plastic  boxes  with  purchase  orders  in  time  for  use,  and,  where 
possible,  storing  materials  from  one  semester  to  another  and  facilitating  reuse  . 
Obtaining  and  housing  live  animals  has  generally  been  the  responsibility  of  educators 
with  the  expertise  to  manage  animals  responsibly  and  ethically. 
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TECHNOLOGY 

With  technology  playing  a  large  role  in  the  development  of  21st  Century  Skills,  YES 
Program  managers  and  educators  give  careful  consideration  to  the  types  and  range 
of  technology-focused  experiences  needed  by  teens. 

Unlike  some  of  their  schoolmates  from  higher -income  families,  YES  Teens  may  not 
have  the  ongoing,  sustained  access  to  laptop  and  netbook  computers  they  need  to 
develop  skills  for  college  and  the  workplace.  Identical  sets  of  laptops  and  netbooks 
are  centrally  managed  in  the  program  and  assigned  for  use  to  STEM  Components 
and  College  Prep  to  maximize  access  and  use  of  computers  in  learning.  Locked 
cabinets  are  available  in  each  classroom  to  store  computers,  along  with  power  cords 
and  extra  batteries  that  may  be  required.  Teens  use  computers  for  research  in 
inquiry-based  and  project-based  learning,  to  blog  on  the  YES  website,  to  employ 
specialized  software  (e.g.  Kahn  Academy),  to  manage  projects,  and  to  develop 
presentations.  Teens  use  general  office  software  packages  as  well  as  more  specialized 
software  needed  in  engineering  and  chemistry  components. 

In  addition,  STEM  Components  such  as  SciJourn,  a  grant-based  component  offered 
in  collaboration  with  the  University  of  Missouri  -  St.  Louis,  introduced  teens  across 
the  program  to  specialized  software  for  digital  storytelling  as  well  as  video  cameras 
and  recorders.  Once  items  such  as  these  were  purchased,  they  were  added  to  the 
inventory  and  managed  by  a  program  manager.  Items  were  checked  in  and  out  to 
minimize  their  need  for  maintenance  and  leverage  their  use  across  areas. 

Finally,  some  components  require  specialized  laboratory  equipment  such  as 
thermometers,  heating  devices,  wave  form  monitors,  power  tools,  and  solar  panels. 
These  items  are  also  inventoried  and  reused,  when  possible. 


SPACE 

Both  location  and  design  of  learning  space  plays  an  important  role  in  the  YES 
Program.  The  YES  Program  began  in  1998  with  learning  and  lab  warehouse  space  a 
block  from  the  main  campus  of  the  SLSC.  Later,  the  lease  and  renovation  of  the 
Taylor  Community  Science  Resource  Center  provided  a  more  flexible  and  better- 
designed  learning  environment  with  more  space,  variable  sized  classrooms,  and  a 
wet  lab.  In  addition,  the  Taylor  Center  offers  office  space  for  staff  members  and 
storage  space  for  materials  and  technology. 

Many  teens  use  public  transportation  to  get  to  the  SLSC,  and  the  Taylor  Center  is 
just  across  the  street  from  a  bus  stop  and  three  blocks  from  a  MetroLink  station 
(St.  Louis’s  light-rail  system).  Since  teens  are  paid  as  employees,  they  cover  their 
own  transportation  costs  as  part  of  their  experiences  in  managing  their  money.  This 
means  bus  passes  are  needed  only  for  teens  in  the  New  Teens  Lab. 

Entering  the  Taylor  Center,  teens  find  a  large,  sunny  open  space  where  they  can  meet 
with  friends,  eat  their  lunch,  and  play  a  game  of  chess.  A  small  kitchen  with  a 
refrigerator  and  well-stocked  cabinets  provides  breakfast  cereals  and  healthful 
snacks.  The  Taylor  Center  includes  two  large-sized  meeting  rooms  (Jolly  North  and 
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Jolly  South),  which  can  be  used  for  large  group  sessions  such  as  Town  Hall  Meetings 
and  New  Teens  Labs.  Smaller  spaces  house  Learning  Labs.  A  wet  lab  provides 
specialized  space  and  equipment  for  some  experiences.  The  building  includes 
private,  keyed  space  for  staff  member  offices,  copiers,  and  private  meeting  rooms. 
Storage  areas  allow  management  of  materials  and  technology.  The  Taylor  Center  has 
two  open-access  WiFi  feeds,  allowing  teens  to  use  web-based  resources  throughout 
the  building. 


STAFF 

Well-qualified  and  committed  staff  members  make  for  a  successful  program.  During  the 
Community  STEM  Outreach  Project,  evaluators  documented  the  impact  of  staff 
reduction  and  turnover.  Yet,  through  changes  in  leadership  and  loss  of  colleagues,  the 
YES  staff  worked  to  keep  the  program  operating  and  adopted  sustainable  practices.  The 
number  of  YES  Teens  served  was  reduced  as  a  result  of  staffing  cuts.  At  the  end  of  the 
funding  period  in  September  2013,  the  program  had  two  managers,  one  support  staff 
member,  and  nine  educators.  During  the  2013  summer,  three  additional  educators  and 
10  YES  Junior  Interns  (YES  Teens)  were  employed  to  manage  the  extended  program 
hours. 

YES  managers  and  educators  were  recruited  to  the  program  from  a  variety  of  fields, 
reflecting  the  research-based  foundations  of  the  program  design  and  practices.  Some 
educators  have  degrees  in  social  work  and  experience  in  youth  development.  Others 
come  into  the  program  from  STEM  research  labs  with  engineering  degrees;  still  others 
bring  degrees  and  experience  from  other  formal  and  informal  education  settings. 

Other  SLSC  departments  provide  the  staff  for  services  to  support  the  program.  The 
Human  Resources  Department  works  closely  with  the  YES  Program.  They  help  with 
hiring  processes  and  in  payment  of  teens,  meeting  directly  with  teens  to  orient  them  to 
workplace  policies  and  advising  them  on  setting  up  bank  accounts  for  direct  deposit  of 
their  wages.  Building  Services  provides  a  security  guard  at  a  desk  at  all  times  the  Taylor 
Center  is  open.  As  required  by  programs  involving  young  people,  visitors  are  signed  in 
and  out,  and  the  guard  is  on  hand  to  assist  in  any  emergencies.  This  department  also 
provides  maintenance  and  janitorial  services.  Information  Systems  purchases  office 
computers  through  its  budget,  provides  advice,  and  maintains  the  building’s  WiFi 
system.  In  addition,  the  Exhibitions  Department  has  collaborated  with  the  YES 
educators  in  developing  and  offering  a  STEM  Component. 


BUDGET 

The  YES  Program  budget  is  part  of  the  Community  Science  Outreach  Department, 
which  also  includes  other  services  such  as  specialized  school  programs  and  teacher 
training  connected  to  the  special  expertise  of  the  department  staff.  In  general,  the  YES 
Program  expenses  include  funds  for  (1)  Salaries  and  Wages;  (2)  Materials  and  Supplies; 
and  (3)  Equipment. 

Salaries  and  Wages  make  up  the  largest  direct  expense  category  for  the  YES  Program. 
This  category  includes  salaries  for  managers,  educators,  and  support  staff  members  as 
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well  as  wages  for  part-time  educators.  In  addition,  as  part  of  the  Work-based  Program 
Learning  strategy,  teens  are  paid,  starting  at  minimum  wage  and,  in  a  few  instances,  at  a 
slightly  higher  level  during  internships. 

Some  program  expenses  are  part  of  the  budgets  of  other  SLSC  departments.  These 
expenses  include  Building  Services  (security,  maintenance,  janitorial);  Information 
Systems  services  and  capital  expenditures;  and  Human  Resources  support  for  hiring 
and  managing  payment  of  staff  as  well  as  insurance  and  other  indirect  costs. 


ONGOING  FUNDING 

Funding  sources  for  the  YES  Program  have  varied  over  the  life  of  the  program.  Through 
the  Saint  Louis  Zoo-Museum  District  (ZMD),  the  SLSC  receives  some  property  tax 
support  from  residents  of  St.  Louis  City  and  St.  Louis  County.  Yet  this  support  provides 
only  about  50%  of  the  funds  necessary  to  carry  operate  the  Science  Center.  Additional 
earned  revenue  through  membership,  special  exhibitions,  theater,  food  service,  and 
parking  are  required.  While  the  YES  Program  staff  has  offered  specialized  teacher  PD 
and  family  workshops  to  earn  revenue,  these  efforts  are  difficult  to  maintain,  given  the 
overlap  with  other  SLSC  department  missions  and  variable  funding  availability  among 
partners  scheduling  the  programs.  Some  state  monies  are  available  to  the  SLSC,  with 
some  set  aside  provide  ongoing  support  for  the  YES  Program. 

Additional  support  has  been  obtained  through  grant-based  funding.  This  funding  has 
brought  with  it  new  ideas  and  expertise  as  well  as  national  partners.  Yet  it  has  the  risk  of 
new  innovations  and  ideas  not  being  sustainable  after  funding  ends. 

Levels  of  available  resources  provide  boundaries  for  the  program— that  is,  the  number  of 
teens  the  program  can  serve  and  provides  a  context  for  the  consideration  of  learning 
strategies  and  STEM  topics.  Trying  to  serve  too  many  teens  with  too  few  resources 
weakens  the  program.  Yet  serving  too  few  teens  may  make  the  program  unattractive  to 
prospective  funders  or  institutional  leaders. 


RESOURCES  AND  PROGRAM  SIZE 

From  1998  to  2012,  the  YES  Program  continued  to  grow,  serving  a  larger  and  larger 
number  of  teens,  hiring  more  staff,  and  spending  more  in  direct  expenses.  Given  the 
national  and  international  economy,  the  YES  Program  is  currently  involved  in 
developing  as  a  program  that  serves  a  sustainable  number  of  teens  with  consistent  and 
sustainable  expenses  and  budget. 

The  YES  staff,  along  with  the  SLSC  Development  Department,  continues  to  develop 
grant-based  funding  ideas  and  projects,  yet  these  funds  are  not  as  widely  available  as  in 
previous  years.  Some  funding  sources  were  affected  by  the  merger  of  education 
functions  across  government  agencies  into  the  U.S.  Department  of  Education.  Yet,  OST 
programming  remains  a  promising  practice,  and  the  YES  Program  with  a  reputation  of 
collaboration  and  innovation— will  continue  to  work  in  these  areas,  not  just  for  funding 
but  as  an  active  partner  in  developing  the  best  practices  and  participating  in  the 
research  that  enriches  this  field. 
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In  addition,  the  YES  Program  continues  to  explore  other  funding  opportunities.  For 
example,  some  Community  Partners  may  be  well-suited  to  assume  interns’  wages  for  the 
teens  working  in  their  businesses,  laboratories,  and  organizations.  Private  donors  may 
be  solicited  to  provide  YES  Program  scholarships.  The  positive  responses  to  the  teen 
performance  make  this  funding  source  more  likely. 


PROGRAM  DOCUMENTS 


YES  Functional  Organization  Chart-2013 
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TARGET  AUDIENCE 


Low-income  and  underserved  St.  Louis  area  teens  between  ages  14  and  18  comprise  the 
target  audience  of  the  YES  Program.  The  program  serves  substantial  numbers  of  girls 
and  members  of  minority  ethnic  groups.  In  addition  to  the  characteristics  shared  by  all 
adolescents,  these  teens  may  have  personal  and  social  issues  in  their  lives  that  provide 
obstacles  to  STEM  education  and  careers.  Demographic  (age,  grade  level,  gender, 
residence)  and  psychographic  descriptions  (courses  taken,  career  plans,  attitudes  as 
they  enter  the  program)  provide  information  about  which  learning  strategies  and 
program  structures  staff  members  should  use  to  support  learning  and  development. 

In  addition  to  the  general  characteristics  of  this  target  audience  presented  in  the 
Learning  Strategies  section  as  the  7  Developmental  Needs  of  Youth,  ongoing  program 
records  and  evaluation  provide  additional  information  about  the  characteristics  of  YES 
Teens.  Using  the  categories  recommended  by  Lynn  Dierking  (p.  28)  in  Framework  for 
Evaluating  Impacts  of  Informal  Science  Education  Projects,  evaluators  of  the 
Community  STEM  Outreach  project  described  the  audience  of  the  YES  Program  in 
terms  of  both  demographic  and  psychographic  characteristics,  providing  a  picture  of  the 
target  audience  for  the  program  staff  members  and  for  other  stakeholding  groups  such 
as  funders,  Community  Partners,  parents,  and  teens. 

The  data  described  in  this  section  of  the  documentation  are  adapted  from  the 
Community  STEM  Outreach  Summative  Evaluation  Report  (Klein  &  Tisdal,  2013).  The 
graphs  and  tables  were  based  on  the  438  teens  participating  in  the  YES  Program  from 
October  2010  through  August  2013.  In  collecting  and  reporting  information,  it  is 
important  to  clearly  define  precisely  the  group  included  in  the  data  presented.  We 
needed  to  define  what  we  mean  by  a  participating  teen.  Working  with  YES  staff 
members,  we  defined  a.  participating  teen  as  a  teen  who  attended  at  least  two  days  in 
any  of  the  eight  semesters  since  the  beginning  of  the  Community  STEM  Outreach 
project  (Spring,  Summer,  and  Fall  2011  and  2012,  plus  Spring  and  Summer  2013). 
Throughout  this  section,  unless  stated  otherwise,  we  used  438  as  the  number  of  teens 
(i.e.,  N  =  438)  in  all  tables  and  graphs. 
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DEMOGRAPHICS  OF  TEENS  IN  THE  YES  PROGRAM 

Tables  1  and  2  and  Figures  3  and  4  show  demographic  data  for  teens  who  participated  in 
the  YES  Program  between  2011  and  2013.  While  there  was  a  balance  between  males  and 
females  in  the  program,  ethnicity  remained  predominately  African  American. 


Female 

Male 

53% 

47% 

N  =  438 

Table  1.  Gender  of  Teens 


Gender 

Total 

Female 

234 

Male 

204 

Total 

438 

Figure  3.  Gender  of  Participating  YES  Teens 


Table  2.  Ethnicity  of  Teens 


Ethnicity 

Total 

Black  or  African 

American 

382 

White 

10 

Asian 

8 

Hispanic 

4 

Multiple  Races 

1 

Declined  to 

Provide  Data 

33 

Total 

438 

Teens  self-reported  ethnicity  data  using  a  wide  variety  of  self-determined  categories.  To 
keep  the  data  in  Figure  4  simple  and  easy  to  read,  evaluators  grouped  teen  self-report 
data  into  the  U.S.  Census  categories.  Because  data  spanned  three  years,  grade  level  data 
are  indicated  by  high  school  graduating  class.  Youth  can  join  YES  as  early  as  age  14, 
which  means  YES  Teens  can  be  in  grades  6-12.  Figure  5  shows  the  number  of  teens 
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participating  by  grade  level  over  the  past  three  years.  Numbers  do  not  track  over  time 
since  incoming  New  Teens  may  be  in  any  grade  level.  The  patterns  in  Figure  5  also  show 
teens  that  may  have  had  another  job,  a  family  issue,  or  involvement  in  school  or 
community  activities  that  prevented  participation  in  a  YES  Learning  Lab  during  one 
year,  even  though  they  returned  to  the  program  to  participate  in  the  next. 


Middle  Freshman  Sophomore  Junior  Senior 
School 


Grade  Level 


a 2010-2011 
b 201 1-201 2 
B 2012-201 3 


Figure  5.  Grade  Levels  of  YES  Teens  in  2010-2011  (N  =  300),  2011-2012  (N  =  290),  and  2012- 
2013  (N=256) 

Table  3.  Grade  Levels  of  YES  Teens  by  Year  of  Project 


2010-2011 

2011-2012 

2012-2013 

Middle  School 

62 

36 

31 

Freshman 

73 

79 

46 

Sophomore 

50 

79 

68 

Junior 

59 

45 

72 

Senior 

56 

51 

39 

Total  (N) 

300 

290 

256 
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YES  Teens  attended  a  wide  variety  of  schools— public,  private,  homeschool,  and  others, 
in  Missouri  and  Illinois.  Since  some  of  the  teens  change  schools  frequently,  data  in 
Figure  6  were  based  on  the  last  reported  high  school  or  middle  school  attended. 
Systematic  collection  of  this  information  shows  that  many  YES  Teens  move  frequently 
and  attend  several  different  high  schools.  Of  the  438  participants,  only  415  provided 
information  about  the  school  they  attended  on  their  program  application  or  on 
questionnaires. 


Figure  6.  Number  of  Participating  YES  Teens  by  School  Type 

While  YES  Teens  attended  a  wide  variety  of  schools,  the  program  drews  substantially 
from  several  public  school  districts  in  the  St.  Louis  area.  Table  4  shows  the  six  school 
districts  from  which  over  50.0%  of  YES  participants  are  drawn.  These  percentages  are 
based  on  the  total  number  of  participants.  All  these  high  schools  have  graduation  rates 
below  the  Missouri  state  average. 

Table  4.  Six  School  Districts  Totaling  over  50.0%  of  Participating  Teens  (AT  =  438) 


School  District 

Percent 

St.  Louis  City 

31.1% 

Hazelwood 

7-5% 

Ferguson-Florissant  R-II 

4.8% 

Normandy 

3.9% 

Parkway  C-2 

3.7% 

Ritenour 

3-0% 
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PSYCHOGRAPHICS  YES  TEENS 

Psychographic  data  provide  a  picture  of  the  importance  of  paid  work  as  a  motivation  for 
teens  joining  and  staying  in  the  program.  As  109  New  Teens  entered  the  program  in 
spring  2011,  they  were  asked  why  they  joined  the  YES  Program.  Ninety-five  responded 
to  the  survey.  The  survey  asked  New  Teens:  What  is  your  main  reason  for  wanting  to 
join  the  YES  Program?  Many  teens  selected  more  than  one  response.  Figure  7 
summarizes  their  responses. 


co 

in 

> 

■o 

CD 

c 

'o 


CO 

c 

CD 

<D 


5 

CD 


Wanted  Work  Experience 
Needed  a  Job 
The  Money 
Family  Wanted  Me  to  Join 
Love  Science 
Friends  Said  It  Was  Fun 
Laptop 

Mentors  Wanted  Me  to  Join 
Other 


>» 

.c 

5 


Number  of  Responses 


Figure  7.  New  Teens’  Reasons  for  Joining  YES 

In  the  spring  of  2013,  returning  YES  Teens  were  asked  why  they  kept  coming  back  to  the 
YES  Program  ( N  =  122).  Again,  they  were  asked  to  check  one  of  the  following  options, 
but  many  selected  several.  Figure  8  summarizes  their  responses. 


Like  Work  Experience 

c 

3  Need  the  Job/Money 

Jj;  Like  Teaching  Kids 

c 

®  Love  Science 

^  My  Family  Makes  Me 
Other 


0  to  20  30  40  50 


Number  of  Responses 


Figure  8.  YES  Teens’  Reasons  for  Returning  to  YES 
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JOURNALING  AND  RELATIONSHIPS 

Research-based  characteristics,  demographics,  and  psychographics  provide  an 
important  overview  of  teens  served  as  a  whole.  Yet  many  teens  have  specific 
characteristics  and  needs  that  the  program  must  address  for  the  teen  to  be  succcessful. 
The  description  of  the  Journaling  Learning  Strategy  included  in  this  document  shows 
how  educators  learn,  in  a  non-intrusive,  voluntary  way,  what  is  going  on  in  teens’  lives 
that  indicate  individual  needs  and  program  adaptations.  Casual  conversations  and  the 
development  of  communications  between  teens  and  staff  members  and  between  the  YES 
Program  and  Community  Partners  also  provide  another  source  of  important 
information  about  the  audience  for  which  the  program  is  designed. 

DELIBERATE  DESIGN 

The  deliberate  design  of  the  YES  Program  is  largely  based  on  a  rationale  focusing  on 
target  audience  needs  and  characteristics.  For  example: 

•  The  7  Development  Needs  of  Youth  is  used  to  provide  appropriately  shaped 
learning  experiences  learning  experiences  throughout  the  program. 

•  The  work-based  program  is  based  on  ongoing  information  about  the  effect  of  this 
program  element  in  helping  to  recruit  and  retain  YES  Teens. 

•  Information  about  the  high  schools  YES  Teens  attend  provides  evidence  about 
the  graduation  rates  of  their  friends  and  schoolmates  and  supports  the  need  for 
College  Prep  Learning  Labs. 

RESULTS 

The  following  quotes  come  from  Community  Partners  describing  the  results  of  reaching 
out  to  the  YES  target  audience.  The  first  comes  from  a  parent  who  volunteers  with  the 
YES  Teens  and  works  for  one  of  the  Community  Partners.  She  describes  the  importance 
of  reaching  the  YES  Program’s  target  audience  with  insight  that  only  a  parent  can  share. 

I’m  also  involved  in  a  parental  role  as  a  YES  Teen’s  parent,  so  I  come  and 
volunteer  at  anything  she  would  come  to.  But  also  what  has  helped  me,  my 
daughter  wants  to  be  a  nurse  or  she  wants  to  be  an  engineer.  So  the  Science 
Center  has  helped  me  to  help  her  realize  she  can  do  biomedical  engineering.  And 
so  there  are  several  YES  Teen  supervisors  who  are  in  engineering  programs,  so 
they’re  teaching  me  how  to  teach  her  so  it’s  like  it’s  a  teachable  moment.  They 
always  say  parents  need  to  teach  themselves.  If  she  sees  her  mom  is  just  as 
involved  in  a  career  path  that  she  wants  to  be  in,  she’s  becoming  more  excited. 
So  it’s  helping  me  to  learn  more  and  I’m  sharing  with  her,  her  friends,  my 
nephew  who’s  in  the  program,  and  several  of  my  clients  [who  are  in  the 
program],  too.  So  what  I’m  saying  is  that  if  it  wasn’t  for  the  YES  Program  I 
probably  would  have  told  her,  I  don't  know  what  biomedical  is,  I  don’t  know 
nothing  about  nursing.  But  because  I’m  actually  trying  to  learn  and  teach  her 
it’s  been  very,  very  helpful.  (May  2012  Community  Partner  Focus  Group) 
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An  educator  described  a  similar  benefit. 


I  feel  like  the  Science  Center  has  helped  develop  these  kids  into  capable  children, 
capable  adults.  Children  that  feel  very  competent  in  who  they  want  to  be. 
Opposed  to  when  they  first  started.  So  now  I’m  hearing  someone  who  says  I 
can’t  stand  science  say  I  want  to  be  a  scientist,  I  want  to  be  a  biomedical 
engineer  because  they  make  $80,000  starting  off.  But  just  knowing  and  feeling 
more  career-focused  and  driven  and  being  capable  is  an  awesome  thing. 
Especially  in  the  African-American  community,  which  is  a  major  population. 
Because  a  lot  of  our  children  lack  confidence,  have  their  heads  down,  don’t  know 
where  they’re  going,  don’t  know  what  they  want  to  be  because  they  haven’t  seen 
the  role  models.  But  they  see  it  here  at  the  Science  Center  and  in  the  Community 
Partners  as  well. 
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PROGRAM 


This  section  provides  an  overview  of  the  program 
as  it  was  intended  to  work  at  one  time,  during  the 
2011-2012  and  2012-2013  program  years.  Even 
within  this  time  frame,  the  program  was  constantly 
changing,  sometimes  in  response  to  external  events 
and  sometimes  as  the  managers  and  educators 
identified  improvements  from  research,  evaluation, 
and  reflective  practice.  Based  on  its  past  history, 
the  YES  Program  will  continue  to  change  and  it 
should.  New  research  on  learning  will  be  released. 

Best  practices  from  other  organizations  will  be 
adopted.  Managers  and  staff  members  with 
innovative  new  ideas  will  contribute  to  the 
program. 

In  addition,  programs  are  dynamic  systems  that  should  respond  to  the  changing  needs 
of  the  communities  and  institutions  in  which  they  exist  and  the  audiences  they  serve. 
Information  included  in  the  model  reflects  (1)  elements  that  appeared,  based  on 
evaluation  findings,  most  influential  in  accomplishing  program  impacts,  and  (2)  ways  of 
understanding  those  elements  that  would  be  the  most  useful  to  audiences  who  wanted 
to  design  a  new  youth  program  or  improve  an  existing  one. 

The  Program  Section  includes  information  about  Learning  Strategies,  Program 
Structure,  and  two  types  learning  labs  used  in  the  YES  Program:  Components  (New 
Teens  Lab,  Summertime  Science,  STEM  Components,  Internships),  and  College  Prep. 

Evaluators  noted  that  one  way  the  YES  Program  has  evolved  over  the  past  three  or  four 
years  is  a  change  in  language  and  thinking  about  New  Teens,  Summertime  Science,  and 
STEM  Components,  and  the  expansion  of  types  of  meaningful  work.  When  the  YES 
Program  began,  teaching  featured  as  the  meaningful  work  in  all  components  and  for  all 
levels  of  program  experience  (i.e.  length  of  time  teens  had  been  in  the  program).  The 
term  Summertime  Science  referred  to  all  summer  STEM  Components  and  where 
younger  children  from  Community  Partners  and  Science  Center  visitors  facilitated 
inquiry-based  STEM  activities.  Each  of  these  components  focused  on  different  STEM 
areas;  for  example,  astronomy,  paleontology,  chemistry,  or  engineering.  Other  teens 
taught  offsite  through  a  program  called  Science  on  the  Go! 

In  summer  2013,  Summertime  Science  was  the  name  of  one  STEM  Component  to  which 
all  members  of  the  new  cohort  were  assigned.  In  previous  years,  staff  members  used  the 
term  New  Teen  only  during  the  spring  semester  as  the  new  cohort  participated  as 
volunteers  in  New  Teens  Lab.  In  previous  years,  the  summer  schedule  featured  several 
components  in  which  the  new  cohort  and  returning  teens  participated  together.  In 
summer  2013,  some  of  these  STEM  Components  for  returning  teens,  such  as  Mystery  of 
Matter,  centered  on  teaching  as  the  work  carried  out  by  teens.  Other  components  for 
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returning  teens,  such  as  Agriscience  and  Exhibit  Lab,  featured  project-based  learning  as 
the  work  carried  out  by  teens.  In  2013,  internships,  an  ongoing  element  of  the  YES 
Program  for  several  years,  were  spread  throughout  onsite  components  and  through  a 
component  called  YES  in  the  Community.  In  some  community-based  internships,  teens 
taught  younger  children,  and  in  others,  teens  performed  other  types  of  work  (e.g., 
working  in  a  research  lab,  performing  office  work). 

LEARNING  STRATEGIES 

Learning  Strategies  are  the  overarching  approaches  to  the  design  and  planning  of 
learning  experiences  used  in  the  YES  Program.  Staff  members  draw  on  strategies  that 
have  emerged  from  research  and  practice.  While  many  learning  strategies  are  employed, 
these  are  the  most  salient  and  serve  as  the  foundation  for  additional  strategies  and 
applications. 

•  Work-based  Program 

•  Teaching  to  Learn— Learning  to  Teach 

•  Journaling 

•  Meeting  the  7  Developmental  Needs  of  Youth 

•  College  and  Career  Readiness— Developing  21st  Century  Skills 

•  Project-based  and  Inquiry-based  STEM  Learning 

•  Welcoming  and  Safe  Environment  for  Identity  Development 

•  Teens  Acculturated  into  STEM  Practice 

•  Reflective  Practice 

•  Role  Models  and  Mentors 

Managers  use  these  Learning  Strategies  to  make  decisions  about  the  nature  and 
structure  of  the  program.  The  strategies  also  inform  their  decisions  about  the  topics  of 
STEM  Components,  recruitment  processes,  and  the  selection  of  Community  Partners. 

As  educators  prepare  for  a  semester  with  the  teens  in  the  YES  Program,  they  draw  on 
learning  strategies  in  the  design  of  curriculum  and  selection  of  activities. 

Based  on  evaluation  findings,  these  learning  strategies  appear  to  be  important  elements 
that  support  youth  accomplishing  the  program  impacts.  When  staff  applied  these 
strategies  consistently,  and  as  intended,  teens  learned  and  accomplishment  followed; 
when  they  did  not,  gaps  in  outcomes  were  observed.  This  quote  from  one  YES  alumnus 
in  the  2013  pilot  alumni  survey  gives  insight  into  the  impact  of  these  strategies. 
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The  Design-It!  component  really  made  me  realize  how  much  of  a  knack  I  had  for 
engineering.  By  giving  us  miscellaneous  materials  to  work  with  and  meaningful 
problems  to  solve,  I  was  able  to  make  discoveries  I  wouldn't  have  the 
opportunity  to  make  in  an  ordinary  classroom  environment.  (2006-2011 
cohort,  African-American  male) 

Each  of  the  strategies  are  as  part  of  the  Deliberate  Design  of  Learning  Labs  Activities 
and  practices  need  to  be  intentional,  purposeful,  and  part  of  the  design  of  an  overall 
learning  system  to  be  effective.  These  are  described  for  each  of  the  Learning  Strategy. 

Program  documents  provide  important  tools  for  program  consistency  and  replication. 
Examples  of  program  documents  include  application  forms,  sign-in  sheets,  curriculum 
plans,  handouts,  and  checklists  used  in  the  YES  Program.1  Program  documents  are 
provided  are  each  Learning  Strategy. 

Many  elements  of  the  YES  Program  are  based  on  research  studies,  reports,  and  articles 
about  best  practices.  When  specific  sources  are  used  in  YES,  these  references  are  cited  in 
the  documentation. 


I  MEETING  THE  7  DEVELOPMENTAL  NEEDS  OF  YOUTH 

I  THE  BIG  IDEA 

Research  in  developmental  psychology  focuses  on  the  needs  individuals  have  to 
progress  from  one  life  stage  to  another.  Adolescents  have  specific  needs  that  must  be 
met  for  individuals  to  successfully  move  through  this  stage  and  develop  into  young 
adults. 


|  DESCRIPTION  OF  USE 

As  part  of  the  design  and  implementation  of  all  Learning  Labs  and  interactions  with 
teens,  YES  educators  explicitly  address  seven  developmental  needs  of  youth.  This 
particular  set  of  needs  (Collins,  1998)  has  been  adopted  by  numerous  youth  programs. 
They  include: 

•  Mastery  and  Achievement 

•  Physical  Activity 

•  Self-definition 

•  Creative  Expression 

•  Positive  Social  Interactions  with  Peers  and  Adults 

•  Meaningful  Participation 

•  Structure  and  Clear  Limits 


1  Copies  of  program  documents  are  not  included  in  this  documentation;  they  are  available  on  the 
website  at  www.yescirclesofsupport.com. 
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The  YES  Curriculum  Template  focused  on  these  needs  in  planning  through  a  section 
that  asks  YES  Educators  to  explain  how  these  needs  are  addressed  in  the  Learning  Lab 
they  will  be  teaching. 

The  overall  design  of  the  YES  Program  also  addresses  the  7  Developmental  Needs  of 
Youth.  The  need  for  Mastery  and  Achievement  is  the  focus  of  project-based  learning. 
Educators  respond  to  teens’  need  for  Physical  Activities  by  including  active  icebreakers 
(games  and  group  challenges)  or  scheduling  field  trips  and  outdoor  activities.  The 
process  of  teens  selecting  STEM  Components  that  reflect  their  own  interests  is  based  on 
the  need  for  Self-Definition.  Project-based  learning  also  responds  to  the  need  for 
Creative  Expression  by  allowing  teens  to  create  their  own  projects  within  a  defined 
problem  space.  The  program  fosters  Positive  Social  Interactions  with  Peers  and  Adults 
in  numerous  ways,  including  the  focus  on  Role  Models  and  Mentors,  building  teen- 
educator  relationships  through  Journaling  and  through  Networking  Events,  where  teens 
meet  and  talk  with  representatives  from  Community  Partner  organizations  about  their 
careers.  The  need  for  Structure  and  Clear  Limits  is  addressed  through  Standard  YES 
Elements  that  that  are  part  of  all  Components  (i.e.,  New  Teens  Lab,  Summertime 
Science,  and  STEM  Components). 

These  Standard  YES  elements  provide  a  consistency  and  structure  across  all  these 
components: 

•  Word  of  the  Day  (WOD) 

•  Quote  of  the  Day  (QOD) 

•  Agenda 

•  Brainteasers/Icebreakers 

•  Journaling 

•  Breaks 


|  DELIBERATE  DESIGN 

Using  the  7  Developmental  Needs  of  Youth  as  a  shared  framework  for  program  design 
and  planning  serves  several  purposes: 

•  Provides  a  sound  research-based  strategy  to  help  teens  move  successfully  from 
adolescence  to  adulthood. 

•  Gives  YES  educators  a  consistent  language  to  discuss  the  needs  of  teens  and  a 
way  to  think  about  and  plan  experiences  that  are  appropriate  for  teens. 

•  Allows  program  administrators  to  monitor  where  and  how  the  needs  are  being 
addressed  across  different  areas  of  the  program. 


|  RESULTS 

When  learning  experiences  include  ways  to  meet  many  of  these  needs,  youth  thrive 
across  multiple  areas  of  their  lives.  Meeting  these  specific  needs  of  adolescents  also 
provides  a  way  to  design  programs  that  keep  them  interested  and  engaged  across  a  long 
period  of  time.  Among  YES  staff  members  interviewed  for  the  evaluation,  the  key  factor 
supporting  ongoing  teen  participation  in  the  program  and  achievement  was  the 
development  of  relationships. 
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One  Community  Partner  described  changes  he’d  seen  in  one  of  the  YES  Teens  he  had 
referred  to  the  program.  By  offering  positive  social  interactions,  creative  expression,  and 
meaning  participation,  one  teen  developed  communication  skills. 

One  of  the  teens  told  me  that  they  had  to  try  to  create  something  that  would 
cook  but  you  can’t  use  a  regular  oven,  or  tools.  You  had  to  come  up  with 
something,  solar  power— .  And  this  is  from  a  teen  who  doesn’t  necessarily  like 
science.  Also,  she  told  me  about  one  time  they  made  a  catapult,  so—.  And 
another  thing  I  noticed  is  that  she’s  able  to  communicate  better.  She’s  kinda 
quiet  and  reserved,  so  it’s  bringing  her  out.  (May  2012  Community  Partner 
Focus  Group) 


I  PROGRAM  DOCUMENTS 


YES  Curriculum  Planning  Template-Green  Engineering-Summer  2013 
YES  Curriculum  Planning  Template-Mystery  of  Matter-  Fall  2012 
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|  WORK-BASED  PROGRAM 

I  THE  BIG  IDEA 

After  a  semester  of  orientation,  participating  as  volunteers,  YES  Teens  work  as 
employees  of  the  SLSC.  As  employees,  they  wear  uniforms,  clock  in,  set  up  bank 
accounts  to  receive  their  paychecks  as  direct  deposits,  and  have  their  performance 
evaluated  by  their  supervisors  (program  educators)  each  semester. 

|  DESCRIPTION  OF  USE 

New  Teens  arrive  for  their  first  semester  eager  to  learn  more  about  the  YES  Program. 
During  this  unpaid  orientation  over  several  Saturdays  in  the  spring,  the  “New  Teens” 
cover  a  variety  of  STEM  and  work-skills  topics.  For  example,  one  session  focuses  on 
systems  where  teens  learn  about  the  Science  Center  as  a  system  (a  21st  Century  Skills 
goal).  In  other  sessions,  they  learn  clocking  in,  the  importance  of  being  on  time,  proper 
dress  for  a  work  setting,  and  appropriate  work  communication.  As  part  of  the  process  of 
becoming  paid  employees,  they  fill  out  employment  applications  and  obtain  all  proper 
documentation  needed  (e.g.,  a  Social  Security  card,  government-issued  ID,  or  driver’s 
license).  Teens  also  set  up  bank  accounts  so  their  checks  can  be  direct  deposited. 
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Beginning  in  their  first  summer  semester,  teens  are  proud  to  earn  a  paycheck  as 
employees  of  the  SLSC.  They  receive  minimum  wage  for  meaningful  work,  such  as 
teaching  younger  children  from  community-based  organizations  on  visits  to  the  SLSC 
(Summertime  Science),  delivering  programs  to  younger  children  onsite  at  partner 
facilities  (Science  on  the  Go),  and  serving  as  interns  in  STEM  laboratories  or 
workplaces.  Some  assignments  may  be  eligible  for  a  slightly  higher  pay  rate.  Regardless 
of  the  rate,  teens  are  full  employees  of  the  SLSC. 


|  DELIBERATE  DESIGN 

The  work-based  program  serves  four  main  purposes: 

•  Removes  barriers  to  participation  in  a  STEM  education  program  for  teens  and 
their  families— Teens  in  the  target  audience  would  normally  need  to  have 
afterschool  and  summer  jobs.  Attending  an  unpaid  program  could  put  a  burden 
on  the  family  that  would  be  required  to  pay  for  transportation,  food,  and 
uniforms  to  support  the  teen’s  participation.  The  work-based  program  is 
designed  to  make  teen  participation  at  least  revenue  neutral  for  their  families. 

•  Develops  skills  and  understanding  of  the  process  of  applying  for  a  job— Teens 
receive  support  from  staff  in  filling  out  applications,  and  providing  other 
information  that  prepares  them  to  access  other  work  settings. 

•  Provides  culturally  acceptable  ways  to  discuss  behavior  and  communication  in 
dominant-culture  work  places— Topics  such  as  dress,  language,  eye  contact,  body 
stance,  and  respect  for  supervisors,  elders,  and  peers  can  be  directly  discussed  as 
part  of  the  teen’s  role  as  an  employee  of  the  institution.  Discussions  such  as  these 
are  removed  from  the  context  of  ethic  identification  and  culture. 

•  Gives  an  immediate  context  for  learning  about  financial  responsibility  and 
management— Teens  in  the  program  learn  how  to  set  up  a  bank  account,  cash  a 
check,  pay  taxes,  and  budget  their  wages  for  transportation  and  meals  as  part  of 
their  participation  in  the  YES  Program. 


|  RESULTS 

When  teens  are  treated  as  employees  and  earn  a  paycheck  they  open  a  bank  account  and 
learn  to  manage  their  money.  They  also  are  able  to  contribute  to  the  family  income  and 
stay  in  the  program. 

The  YES  staff  members  are  called  Educators  by  the  Human  Resources  staff  but  are 
called  Supervisor  by  the  teens.  They  walk  a  fine  line  between  teaching  and  being  “the 
boss.”  One  Senior  Educator  described  how  she  saw  her  role  in  a  focus  group. 

Well,  I  think  as,  like  when  we  call  ourselves  their  supervisors,  we  have  like  a 
responsibility  to  them  to  help  them  grow,  to  move  forward  on  a  path  that 
they’ve  identified  for  themselves.  And  we  help  them  identify  that  path  for 
themselves,  I  think,  as  opposed  to  telling  them  what  it  should  be,  what  they 
might  hear  from  other  adults  in  their  lives.  I  think  they  have  freedom  to  joke 
around  with  us  and  be  goofy  enough  with  us,  but  to  know  that  we  have  their 
best  interests  in  mind  and  that  we  will  like,  we’ll  be  the  adult  when  we  are  called 
to  be.  (August  2012  YES  Educator  Focus  Group) 
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Community  STEM  Outreach:  Second  Annual  Evaluation  Report  presents  evidence  that 
the  work-based  program  is  the  primary  motivator  for  teen  participation.  The  top  three 
responses  to  a  question  about  why  New  Teens  in  2011  joined  the  YES  Program  focused 
on  the  work-based  program.  The  combined  percentage  of  these  three  responses  was 
59-9%. 


Findings  from  a  pilot  YES  Alumni  Survey  conducted  in  2013  indicate  that  the  work- 
based  program  element  of  the  YES  Program  supported  several  program  impacts. 

Sample  responses  include: 

I  like  to  very  proudly  tell  people  that  I've  been  working  since  the  age  0/13.  In  the 
summer  of  ’98, 1  relied  on  Bi-State  to  transport  me  to  and  from  St.  Louis  Science 
Center  five  days  a  week.  I  learned  to  create  lesson  plans  by  first  testing  out  the 
lesson/practicing  the  skills  we’d  need  to  clearly  communicate  to  our  students.  I 
learned  to  work  diligently  even  if  teammates  were  slacking  off.  I  learned  the 
value  of  teamwork— the  unique  skills  that  each  person  brings  will,  sooner  or 
later,  prove  essential.  I  also  learned  to  call  ahead  for  lunch  orders  because  it 
takes  a  mighty  long  time  to  walk  to  Imo’s  &  back.  While  the  rest  of  my  friends 
watched  MTV  &  soaked  up  mall  air  conditioning,  I  learned  that  earning 
compensation  for  something  you  really  enjoy  outweighs  the  time  lost 
socializing.  YES  was  an  incredible  way  to  begin  building  a  work  history  that 
has  resulted  in  more  than  10  years  in  Program/Curriculum  Design  &  Youth 
Development,  and  the  only  way  I've  been  working  since  the  age  of  13!  If  it 
weren’t  for  YES,  I'd  have  to  find  something  else  to  brag  about.  (1998-2000 
cohort) 

YES  taught  me  how  to  be  a  key  role  player  in  the  work  environment,  it  also 
taught  me  to  never  settle,  there’s  always  room  for  improvement  in  the  work 
area  and  I  can  go  as  far  as  my  dreams.  (2008-2012  cohort) 

It  prepared  me  to  be  able  to  work  with  other  people  and  to  always  be  on  time 
for  work.  (2005-2009  cohort) 


\  PROGRAM  DOCUMENTS 


Application  Form 
Sign-in  Sheets 
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|  TEACHING  TO  LEARN-LEARNING  TO  TEACH 

|  THE  BIG  IDEA 

In  the  YES  Program,  teaching  younger  children  and  members  of  the  general  public  in 
galleries  of  the  Saint  Louis  Science  Center  provides  an  authentic  and  meaningful  context 
for  learning  STEM  concepts,  teens  engage  in  an  iterative  process  developing 
understanding  of  STEM  concepts  at  deeper  and  deeper  levels  as  they  prepare  to  teach, 
consider  how  to  communicate  ideas,  test  activities,  and  reflect  on  their  experiences. 

They  design  and  use  inquiry-based  experiences  to  teach  others,  requiring  them  to  reflect 
on  their  own  learning  processes  as  well  as  that  of  others.  Since  their  understanding  will 
be  tested  in  a  real  situation,  teens  are  motivated  to  monitor  their  own  understanding 
and  revisit  concepts,  refining  and  deepening  their  knowledge. 

Learning  in  Context  for  Meaning  and  Relevance  is  one  of  the  foundations  of  the  YES 
Program.  Teaching  to  Learn— Learning  to  Teach  and  Project-based  Learning  are  two 
learning  strategies  built  on  this  foundational  element. 


|  DESCRIPTION  OF  USE 

During  the  school  year  as  part  of  the  New  Teens  Lab  or  STEM  Components,  teens 
prepare  to  teach  younger  children  from  Community  Partner  programs.  First,  they 
engage  in  inquiry-based  activities  to  become  familiar  with  concepts  they  will  teach 
during  the  following  summer.  Then,  teens  work  in  groups  to  design  and  practice 
presentations,  explaining  the  concepts  in  their  own  words  and  repeating  the  inquiry- 
based  activities  with  the  goal  of  helping  younger  children  learn.  In  this  process  they 
become  aware  of  gaps  in  their  own  understanding  and  revisit  STEM  concepts  to  develop 
a  deeper  understanding.  As  groups  of  YES  Teens  lead  activities  with  younger  children, 
they  repeat  the  information  many  times,  becoming  more  confident  in  their  grasp  of  the 
concept.  They  also  identify  other  gaps  in  their  understanding  as  they  answer  questions 
from  children  and  engage  in  Reflective  Practice  with  members  of  their  group  about  what 
went  well  and  what  could  be  improved  in  facilitation  of  activities. 

For  example,  teens  in  the  New  Teens  Lab  may  do  a  variety  of  activities  on  buoyancy, 
knowing  they  are  going  to  have  to  design  activities  to  teach  this  topic  to  younger 
children  during  Summertime  Science.  During  the  summer,  they  work  in  teams  to  refine 
activities.  They  teach  an  activity,  over  and  over  again,  across  the  course  of  two  or  three 
weeks  with  groups  of  younger  children,  allowing  them  to  practice  information  in  a 
meaningful  context.  During  debriefing  sessions  each  day,  teens  identify  what  went  well 
and  identify  specific  plans  for  improving  things  that  didn’t  go  as  well  as  they  would  have 
liked.  They  identify  questions  from  children  that  were  challenging  to  answer  and  return 
to  the  concept  of  buoyancy  to  learn  it  on  a  deeper  level.  Teens  teach  concepts  again  and 
again  to  multiple  groups  until  they  become  tired  of  the  topic  and  know  it  thoroughly. 
Then  they  move  on  to  other  activities  based  on  a  different  STEM  topic  they  learned 
during  the  New  Teens  Lab. 
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|  DELIBERATE  DESIGN 

Teaching  to  Learn  and  Learning  to  Teach  serves  several  functions  in  the  YES  Program: 

•  Offers  a  framework  for  an  iterative  process  where  teens  can  learn  STEM  concepts 
at  progressively  deeper  and  deeper  levels 

•  Provides  a  meaningful  and  relevant  context  for  learning  that  gives  immediate  and 
authentic  feedback 

•  Builds  teens’  ownership  of  their  own  learning  and  develops  skills  in  identifying 
where  they  have  clear  grasps  of  content  and  where  they  need  more  clarity  (i.e., 
meta-cognition,  the  ability  to  assess  the  status  of  their  own  learning) 

•  Provides  comfortable  and  respectful  ways  for  teens  to  revisit  and  learn  STEM 
concepts  they  may  have  missed  at  a  lower  grade  level  in  school 

•  Develops  teens’  confidence  as  younger  children  look  up  to  them  and  respect  their 
knowledge 

•  Allows  practice  of  public  speaking  and  communication  skills 


|  RESULTS 

Results  of  this  learning  strategy  include: 

•  Deeper  and  more  thorough  understanding  of  STEM  concepts  that  can  be  transferred 
to  school  and  applied  in  their  lives 

•  Improved  communications  skills,  including  vocabulary  and  public  speaking 

•  Increased  skills  in  meta-cognition. 

•  Increased  ownership  of  their  learning 

•  Increased  confidence  and  self-respect  through  authentic  success  in  teaching  younger 
children 

Findings  from  the  2013  pilot  YES  Alumni  Survey  indicate  that  the  Teaching  to  Learn 
and  Learning  to  Teach  element  of  the  YES  Program  supported  several  program  impacts. 
As  one  former  YES  Teen  commented,  “I  learned  to  create  lesson  plans  by  first  testing 
out  the  lesson/practicing  the  skills  we'd  need  to  clearly  communicate  to  our  students.” 
(1998-2000  cohort) 


|  JOURNALING 

|  THE  BIG  IDEA 

Unlike  the  use  of  Journaling  in  some  other  contexts,  YES  Teens  write  in  journals— 
without  cues  or  prompts— about  what  is  going  on  in  their  lives,  as  opposed  to  reflecting 
about  their  program  experiences.  They  begin  their  program  day  writing  journals. 


|  DESCRIPTION  OF  USE 

As  teens  enter  the  room  for  a  component  session,  they  pick  up  their  journals  (spiral 
notebooks  or  composition  books)  from  a  central  table  and  take  a  seat.  Music  plays  from 
a  CD  player  as  the  teens  write  10  to  15  minutes  about  anything  and  everything  that 
affects  their  lives  or  interests  them.  When  the  journaling  time  is  over,  the  educator  turns 
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off  the  CD  player,  asks  for  volunteers  to  collect  the  journals,  and  begins  the  day’s 
activities. 

Later  that  week,  the  educator  reads  each  journal  and  adds  entries  with  individual 
comments.  A  page  folded  over  means  the  teen  decided  she  didn’t  want  her  journal  entry 
to  be  read,  so  the  educator  will  skip  that  entry.  A  note  written  on  a  folded  page  means 
the  teen  wants  a  particular  educator  or  staff  member  to  read  an  entry  and  no  one  else,  so 
the  educator  will  follow  his  or  her  directions. 

|  DELIBERATE  DESIGN 
Journaling  serves  three  main  purposes: 

•  A  learning  tool— Journaling  allows  teens  to  use  and  improve  writing  and 
observation  skills. 

•  An  assessment  tool— Journaling  allows  staff  to  assess  teens’  writing  skills  and  to 
monitor  and  assess  teens’  developmental  needs. 

•  A  relationship  building  tool— Journaling  allows  teens  and  educators  to  develop 
individual  relationships,  as  teens  write  about  the  things  on  their  minds  and 
educators  respond  with  personal  notes. 

To  best  accomplish  these  purposes,  teens  write  in  journals  at  the  beginning  of  the 
Learning  Lab  without  writing  prompt  from  the  educator,  and  educators  write  responses 
in  the  journals,  as  appropriate. 


|  RESULTS  OF  USE 

When  educators  have  teens  write  in  journals  without  specific  writing  prompts  at  the 
beginning  of  their  day  (i.e.,  following  the  Deliberate  Design),  teens  write  about  a  wide 
array  of  topics,  such  as  their  date  to  prom,  a  challenge  with  a  friend,  or  a  challenge  at 
school.  Often,  teens  will  respond  to  the  comments  and  questions  of  the  educator.  These 
written  conversations  allow  educators  to  understand  what  is  going  on  in  each  teen’s  life, 
to  respond  appropriately,  and  to  develop  an  ongoing  relationship. 

When  educators  have  teens  write  in  journals  at  the  end  of  the  day  (not  following  the 
deliberate  design),  journal  entries  often  are  a  recap  of  the  day’s  activities.  When 
educators  give  teens  writing  prompts,  they  limit  the  teens’  writing  and  possible  insights 
educators  may  gain  from  their  teens.  End-of-day  and  cued  journaling  strategies  are  not 
as  effective  in  building  relationships  between  teens  and  educators. 


I  PROGRAM  DOCUMENTS 


YES  Program— Edited  Journal  Excerpt 
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COLLEGE  AND  CAREER  READINESS-DEVELOPING  21ST  CENTURY 
SKILLS 


|  THE  BIG  IDEA 

The  YES  Program  incorporates  College  and  Career  Readiness  across  all  program 
elements  and  uses  this  strategy  as  an  explicit  part  of  the  deliberate  design  of  the 
program  as  a  whole.  Career  and  College  Readiness  focuses  the  development  of  specific 
skills  and  competencies  that  prepare  teens  to  be  successful  in  college  and  in  the 
workplace. 

Career  and  College  Readiness  is  a  concept  that  emerged  from  two  important  U.S. 
government  reports.  Both  reports  stress  the  importance  of  developing  specific 
competencies  that  prepare  youth  for  college  and  for  the  workplace.  One  of  these  reports, 
published  in  1991  by  U.S.  Department  of  Labor  and  referred  to  hereafter  as  the  SCANS 
Report,  was  titled  Skills  and  Tasks  for  Jobs:  A  SCANS  Report  for  America  2000. 
(SCANS  stands  for  the  Secretary’s  Commission  on  Achieving  Necessary  Skills,  referring 
to  the  U.S.  Secretary  of  Labor.) 

Another  report,  21st  Century  Skills  for  21st  Century  Jobs,  released  in  1999  and  referred 
to  hereafter  as  the  21st  Century  Skills  Report,  was  a  joint  publication  of  several  U.S. 
government  departments,  including  the  Department  of  Education.  Both  these  reports 
provide  frameworks  of  skills  to  prepare  youth  for  college  and  careers.  While  these 
frameworks  of  skills  are  similar,  they  list  different  areas  of  competency  that  young 
people  need  to  develop. 

The  SCANS  Report  defines  five  areas  of  competency  for  effective  workers: 

•  Allocation  of  Resources 

•  Interpersonal  Skills 

•  Information  Skills 

•  Systems  Thinking 

•  Technology  Skills 

The  SCANS  Report  also  provides  three  foundational  competency  areas  that  workers 
must  have: 

•  Basic  Skills 

•  Thinking  Skills 

•  Personal  Qualities 

In  the  YES  Program,  many  activities  used  in  the  New  Teens  Lab  and  in  other 
components  are  guided  by  a  publication  from  the  Association  of  Science-Technology 
Centers’  YouthALIVE!  Project.  YouthALIVE!  used  the  SCANS  Report  as  the  basis  for  its 
own  publication. 
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The  2ist  Century  Skills  Report  defined  several  core  academic  subjects  in  which  young 
people  must  develop  competencies  to  be  ready  for  college  and  the  workplace.  These 
included  English,  reading  or  language  arts,  world  languages,  arts,  mathematics, 
economics,  science,  geography,  history,  government,  and  civics.  The  21st  Century 
Skills  Report  also  set  forth  what  they  referred  to  as  21st  Century  interdisciplinary 
themes: 

•  Global  Awareness 

•  Financial,  Economic,  Business  and  Entrepreneurial  Literacy 

•  Civic  Literacy 

•  Health  Literacy 

•  Environmental  Literacy 


|  DESCRIPTION  OF  USE 

In  the  New  Teens  Lab,  activities  focus  on  systems  thinking  as  YES  Teens  learn  about  the 
Science  Center  as  a  system  and  map  another  system  of  their  choice  (school,  sports,  etc.). 
They  develop  interpersonal  skills  and  thinking  skills  as  they  engage  in  icebreakers  that 
require  teens  to  work  together  to  meet  a  challenge.  The  YES  Program  Curriculum 
Template,  used  for  all  STEM  Components,  includes  an  area  for  educators  to  explicitly 
state  when  and  how  they  are  addressing  each  of  the  five  21st  Century  Skills 
interdisciplinary  themes. 


|  DELIBERATE  DESIGN 

Curriculum  plans  for  all  components  require  staff  members  to  specifically  describe  how 
the  21st  Century  Skills  is  addressed  through  the  sequence  and  selection  of  activities  they 
carry  out  with  students. 


|  RESULTS 

Each  component  of  the  YES  Program  engages  teens  in  experiences  to  develop  21st 
Century  Skills.  As  summarized  in  Table  5,  in  the  summer  2013  Exhibit  Lab  component, 
educators  outlined,  in  their  curriculum  plan,  how  the  activities  in  this  STEM  Component 
would  develop  the  following  21st  Century  Skills. 
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Table  5.  21st  Century  Skills  Developed  in  Exhibit  Lab  Summer  2013 


Skills  Category 

Exhibit  Lab  Experience 

Financial,  Economic,  Business,  and 
Entrepreneurial  Literacy 

Teens  will  be  encouraged  to  keep  supplies  to  a 
minimal  cost. 

Civic  Literacy 

Teens  will  be  exposed  in  a  limited  way  to  Science 
Center  hierarchical  exhibit  governing  system. 

Creativity  and  Innovation 

They  will  be  encouraged  to  use  their  creativity  in 
creating  exhibits  that  meet  objectives.  Teens  will  be  in 
constant  discussion  and  revision  of  their  exhibit  ideas. 
Teens  will  be  designing  their  projects  that  visitors  can 
potentially  recreate  at  home.  The  teens  will  be 
encouraged  to  use  recycled  and  common  materials. 

Environmental  Literacy 

4s  part  of  the  exhibit  design,  teens  will  be  using  a 
wide  variety  of  recycled  items.  This  will  naturally  bring 
up  the  subject  of  recycling  and  its  benefits.  This  will 
be  a  springboard  for  casual  yet  vital  discussions  of 
ways  we  can  easily  integrate  found  and  recycled 
objects  into  our  everyday  lives. 

Critical  Thinking  and  Problem  Solving 

Teens  will  need  to  problem-solve  together  as  they 
choose  and  create  effective  exhibit  prototypes.  All 
points  of  critical  thinking  will  be  needed  and  used  in 
this  process  as  they  choose  and  create  effective 
exhibit  prototypes. 

I  PROGRAM  DOCUMENTS 


YES  Program  Curriculum-Exhibit  Lab-Summer  2013 
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|  PROJECT-BASED  AND  INQUIRY-BASED  STEM  LEARNING 

I  THE  BIG  IDEAS 

While  Project-based  Learning  and  Inquiry-based  STEM  Learning  can  be  considered  two 
distinct  approaches,  they  share  the  characteristic  of  being  experiential  based  strategies. 
Experiential  learning  (Kolb,  1984)  involves  learning  through  direct  experience.  In 
addition,  both  approaches  involve  the  application  of  scientific  approaches  where  teens 
experiment  with  real  materials  to  reach  conclusions  about  what  to  think  or  do.  In 
project-based  learning,  teens  apply  the  engineering  design  process  to  design,  test,  and 
redesign  products  to  meet  challenge  specifications.  In  Inquiry-based  STEM  Learning, 
teens  hypothesize,  test,  and  refine  their  understanding  of  science  concepts.  These  two 
approaches  build  on  the  YES  Program’s  foundation  of  allowing  teens  to  learn  about 
STEM  topics  in  context. 


|  DESCRIPTION  OF  USE 

Project-based  and  Inquiry-based  learning  are  featured  in  learning  labs  across  the 
program.  Some  project-based  learning  experiences  frame  the  plans  of  entire  semester 
length  experiences.  For  example,  teens  in  Design  Engineering  focused  most  of  their 
work  in  the  summer  of  2013  designing  and  building  aquaponic  systems.  Teens  in 
Robotics  STEM  Components  in  previous  semesters  built  robots  and  competed  in  the 
FIRST  Competition.  Teens  in  the  Biofuels  Component  spent  their  entire  semester 
designing  sustainable  devices  to  cook  food.  For  example,  Agriscience  teens  planned  the 
menu  and  prepared  for  the  end-of-summer  YES  event,  demonstrating  their  knowledge 
of  healthful  and  local  food  and  using  their  cooking  skills.  Shorter  projects  are  limited  to 
one  or  two  sessions. 

Inquiry  is  a  very  important  part  of  the  YES  Program.  In  the  New  Teens  Lab,  inquiry- 
based  learning  is  introduced  early  through  focusing  on  heat  transfer  in  making  ice 
cream  and  identifying  living  and  non-living  things.  These  early  inquiry  experiences  are 
intended  to  get  teens  excited  about  science  and  to  develop  their  own  inquiry  skills.  In 
STEM  Components,  YES  Teens  may  engage  in  longer  and  more  sophisticated  inquiry 
projects.  In  Agriscience,  teens  developed  their  own  experiments  and  collected  data  to 
explore  the  effect  of  the  levels  of  light  and  water  on  plant  growth.  In  a  Neuroscience 
component,  teens  designed  formal  experiments  to  explore  factors  affecting  rat  learning. 

Much  like  the  YES  Teens  in  Learning  Labs,  when  Community  Partners  attend  the 
monthly  meetings  with  YES  educators,  they  experience  inquiry-based  learning  activities 
and  develop  new  skills  for  working  with  the  children  in  their  own  programs. 
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|  DELIBERATE  DESIGN 

Experience-based  learning  is  particularly  important  for  underserved  teens  who  may 
have  entered  the  formal  education  setting  without  some  basic  life  experiences  that 
textbooks  used  in  formal  education  assume  they  have. 


|  RESULTS 

The  most  important  result  of  these  two  learning  strategies  is  that  teens  who  dislike  or  do 
not  do  well  in  school  in  STEM  courses  find  they  enjoy  STEM  classes  and  can  grasp  the 
underlying  concepts  when  they  have  the  opportunity  to  experience  and  experiment  with 
the  real  phenomena.  In  addition,  Project-based  and  Inquiry-based  experience  support 
the  Acculturation  of  Teens  into  Science  Practice  as  they  use  the  scientific  processes  to 
approach  their  work  as  well  as  specialized  tools  and  materials. 


|  PROGRAM  DOCUMENTS 


YES  Biofuels  Final  Design  Concept  Form 

YES  Design  Engineering -Fall  2013 

YES  Ice  Cream  You  Scream!  New  Teens  Lab 
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2  Downloading  workshop  materials  requires  flash  and  can  only  be  done  in  pop-up  windows.  It 
may  be  a  bit  difficult,  but  the  materials  are  valuable— so  be  patient. 
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|  WELCOMING  AND  SAFE  ENVIRONMENT  FOR  IDENTITY  DEVELOPMENT 

|  THE  BIG  IDEA 

Creating  a  Welcoming  and  Safe  Environment  for  Identity  Development  means  teens  can 
leave  the  baggage  of  problematic  identities  at  the  door  when  they  enter  the  YES  Program 
and  develop  identities  that  will  serve  them  well  in  school,  in  college,  and  in  the 
workplace.  Whether  they  see  themselves  as  behavior  problems,  bad  students,  bad  at 
STEM,  or  are  just  shy  and  find  it  hard  to  make  friends,  they  start  again  in  the  YES 
Program  and  learn  to  take  risks  and  see  themselves  as  successful.  When  a  teen  enters 
the  program,  YES  staff  members  have  no  idea  why  they  were  selected  by  the  Community 
Partner  to  apply.  YES  Teens  do  not  submit  school  grades  to  the  program  when  they  are 
admitted.  YES  Teens  begin  the  program  with  a  clean  slate.  The  YES  Program  provides  a 
safe  place  to  make  take  risks  and  make  mistakes. 


|  DESCRIPTION  OF  USE 

Three  actual  situations  encountered  as  part  the  YES  Program  evaluation  provide 
examples  of  how  the  program  builds  a  safe  and  welcoming  environment  to  give  teens  an 
opportunity  to  develop  positive  identities  that  will  serve  them  well  in  school,  college, 
and  the  workplace. 

One  teen  entered  the  YES  Program  disliking  STEM  courses  in  school  and  fearing 
embarrassment  when  he  didn’t  understand  the  concepts  in  class.  He  was  not 
comfortable  with  the  subject  matter.  He  joined  a  group  of  other  teenagers  in  the  New 
Teens  Lab  and  participated  in  inquiry-based  STEM  activities.  Rather  than  just  talking 
about  living  and  non-living  things,  he  held  live  chickens  and  boxes  of  cereal  and  tried  to 
classify  them  into  groups  with  the  characteristics  defining  the  categories.  Rather  than 
reading  about  buoyancy,  he  developed  a  hypothesis  about  what  would  float,  a  can  of 
regular  soda  or  a  can  of  diet  soda.  He  was  wrong  and  learned  that,  in  science,  being 
wrong  can  be  productive  and  lead  to  more  questions  and  deeper  understanding.  In 
summertime,  he  taught  younger  children  and  led  these  same  activities.  These  younger 
children  from  Community  Partners’  organizations  saw  him  as  a  science  expert,  and  he 
guided  their  inquiry  and  answered  questions.  His  confidence  grew.  Later,  he  shared  his 
summer  experience  at  school  and  teachers  transferred  him  from  lower-level  STEM 
courses  to  the  college-bound  track  because  he  had  taught  science  at  the  Science  Center. 
He  did  well  in  these  courses.  Gradually,  he  began  to  see  himself  as  someone  who  was 
good  at  science,  good  at  math,  and  capable  or  working  in  a  STEM  area.  Currently,  he  is 
an  apprentice  electrician. 

In  another  instance,  a  teen  repeatedly  wore  the  wrong  uniform  pants.  When  questioned 
(one-on-one,  quietly  during  a  break),  the  educator  learned  that  the  teen  had  neither  the 
appropriate  pants  nor  the  money  to  buy  them.  A  pair  of  uniform  pants  was  supplied  by 
the  program.  The  educator  avoided  embarrassing  the  teen  and  used  an  infraction  of  the 
rules  to  get  more  information  and  help  solve  a  problem.  Gradually,  the  teen  began  to  see 
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adults  as  people  with  the  capacity  to  treat  him  with  respect,  listen,  and  help  him  solve 
problems;  for  him,  adults  were  no  longer  just  rule-enforcers  to  fear  and  avoid. 

On  another  occasion,  a  teen  lost  her  temper  and  yelled  at  the  educator  supervising  her 
STEM  Component.  Emotions  flared  on  both  sides,  and  the  supervisor  told  the  teen  to 
leave  and  talk  to  a  manager.  The  supervisor  recognized  that  he  was  not  in  the  position  to 
work  with  the  teen  at  that  moment.  The  manager  sat  with  the  teen  and  calmly  asked, 
“What’s  up?”  Sitting  quietly  and  listening,  the  supervisor  learned  about  all  the  stress 
going  on  in  the  teen’s  life.  After  talking,  the  teen  became  very  apologetic  about  her 
behavior.  The  manager  helped  her  work  out  a  strategy  to  apologize  and  mend  the 
relationship  with  the  supervisor.  The  teen  had  a  safe  environment  where  one  mistake 
did  not  make  her  a  problem.  Rather,  after  the  incident,  the  teen  felt  like  a  person  valued 
and  understood,  capable  of  solving  a  problem. 

These  examples  are  a  few  ways  the  YES  staff  members  build  a  safe  and  welcoming 
environment  where  teens  can  risk  and  learn  from  mistakes  and  develop  new  identities 
that  help  them  in  school,  college,  and  the  workplace. 


|  DELIBERATE  DESIGN 

Providing  a  Safe  and  Welcoming  Environment  for  Identity  Development  serves  several 
purposes  in  the  YES  Program: 

•  Allows  teens  who  have  built  negative  identities  in  other  environments  to  start 
fresh  and  learn  to  see  themselves  and  others  in  new  ways 

•  Provides  ways  for  teens  who  have  already  “dropped  out  of  the  pipeline”  toward 
STEM  education  and  careers  to  move  back  into  that  educational  pathway 

•  Encourages  teens  to  stay  in  the  program  over  a  long  period  of  time 

•  Builds  identities  and  the  problem-solving  skills  teens  need  to  be  successful  in 
other  settings 


|  RESULTS 

A  safe  and  welcoming  environment  has  several  results: 

•  Teens  who  fear  and  avoid  STEM  courses  in  school  gain  confidence  and  see 
themselves  as  students  capable  of  doing  well  in  these  areas. 

•  Teens  who  have  defined  themselves  as  hating  science  or  other  STEM  topics  find 
science  them  interesting  and  engaging,  and  redefine  themselves  as  those  who 
now  like  biology,  physics,  or  engineering. 

•  Teens  who  have  defined  themselves  as  trouble-makers  find  they  enjoy  the 
alternate  identity  of  being  good  team  members  who  cooperating  with  and  lead 
others. 

•  Teens  with  stressful  life  issues  about  which  they  feel  angry  or  helpless  find  other 
ways  to  communicate  safely  about  these  areas  of  life,  and  develop  capabilities  to 
solve  problems  themselves  or  with  the  help  of  others. 

One  Summer  Intern  described  how  she  saw  the  development  of  a  safe  and  welcoming 
environment  in  the  Summertime  Science  program. 
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I  think  when  teens  come  here  to  YES,  it  really  is  very  much  a  community.  We 
talk  about  this  YES  community,  and  I  think  the  community  aspect  of  it  does 
exist.  So  I  think  ...  they  come  to  a  place  where  they’re  able  to,  you  know,  the 
relationship  aspect  has  been  talked  about  a  lot,  but  I  think  it’s  probably 
somewhere  they  know  people  believe  that  they  can  succeed  and  expect  them  to 
succeed.  (August  2012  YES  Summer  Intern  Focus  Group) 
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|  TEENS  ACCULTURATED  INTO  STEM  PRACTICE 

|  THE  BIG  IDEA 

To  be  a  successful  member  of  the  STEM  workforce,  teens  need  to  be  acculturated  into 
STEM  practice.  The  culture  of  STEM  practice  includes  adopting  the  language  and 
behavior  that  are  part  of  this  culture  and  using  them  in  the  appropriate  settings. 
Absorbing  and  being  comfortable  with  the  specialized  vocabulary  of  different  areas  of 
STEM  is  particularly  important,  along  with  the  recognition  that  any  new  area  will  have 
different  concepts,  vocabulary,  and  equipment  individuals  must  learn  and  practice.  In 
addition,  they  learn  and  practice  behaviors  such  as  carefully  making  and  recording 
measurements,  using  specialized  equipment,  keeping  records  of  experiments, 
productively  solving  conflict  among  team  members,  mentoring  younger  team  members, 
and  presenting  results— all  types  of  behaviors  come  into  play  in  the  life  of  an  engineer,  a 
field  biologist,  or  a  physician. 


|  DESCRIPTION  OF  USE 

The  Acculturation  into  STEM  is  a  Learning  Strategy  that  pervades  the  YES  Program. 
From  the  New  Teens  Lab  where  teens  don  safety  glasses  to  mix  chemicals,  to 
community  internships  in  research  labs,  to  STEM  Components  where  teens  keep  science 
journals  about  the  experiences  that  are  part  of  their  inquiry-based  learning,  the  program 
immerses  teens  in  the  cultures  of  various  areas  of  science. 

For  example,  in  the  Biofuels  STEM  Component,  teens  are  challenged  to  design  a  product 
that  will  (1)  decrease  the  energy  needs  of  the  user  or  produce  energy  for  them;  and  (2) 
be  built  with  sustainable  materials.  The  product  design  must  not  exist  before  teens 
create  it.  As  part  of  this  design  challenge,  teens  learn  the  language  of  biofuels  research 
and  engineering,  using  terminology  such  electrical  energy,  diesel,  ion,  magnetism,  and 
gears.  After  introduction  to  science  and  engineering  concepts,  they  begin  their  own 
design  process.  Teens  apply  the  scientific  process,  hypothesizing  and  then  carrying  out 
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tests  and  making  repeated  measures  about  what  happens  with  different  energy  sources 
and  equipment  designs.  Experiments  are  documented  in  a  science  journal.  Teens 
present  progress  reports  to  the  larger  group  and  reflect  on  the  status  of  their  designs 
and  their  own  understanding  of  the  underlying  concepts  and  process.  As  part  of  this 
component,  they  use  the  materials  and  equipment  often  found  in  research  laboratories, 
such  as  vented  hoods,  temperature  measures,  safety  goggles,  and  a  variety  of  other  tools. 
At  the  end  of  these  STEM  Components,  like  professional  researchers  and  engineers, 
they  present  their  designs  to  their  colleagues  and  some  semesters  may  present  their 
designs  to  wider  audiences,  such  as  the  Saint  Louis  Science  Center  Board. 


|  DELIBERATE  DESIGN 

The  purpose  of  this  learning  strategy  in  the  YES  Program  is  to  immerse  teens  in  the 
culture  of  science  through  project-based  and  inquiry-based  learning  to  allow  them  to 
absorb  and  adopt  the  culture  of  different  fields  of  STEM  practice.  This  strategy  prepares 
them  for  other  STEM  educational  experiences  and  provides  an  authentic,  experiential 
view  into  the  work  professionals  do  in  STEM  careers. 


|  RESULTS 

Results  of  this  learning  strategy  include  teens: 

•  Recognizing  that  entering  any  new  area  of  STEM  practice  requires  learning  the 
language  and  how  to  use  tools  and  equipment 

•  Developing  an  appreciation  for  the  need  to  work  in  a  systematic  way  and  carefully 
recording  measurements  and  observations. 

•  Understanding  that  scientists  and  engineers  must  clearly  present  their  work  to 
colleagues  to  communicate  their  findings 


I  PROGRAM  DOCUMENTS 


YES  Agriscience  Curriculum  Planning  Template-Summer  2013 
YES  Biofuels  06  Making  Biodiesel  Activity 
YES  Biofuels  Component  Spring  2013  Project  Design 
YES  Biofuels  Fall  2012  schedule 

YES  Reflection  and  Evaluation  Energy  and  Biofuels  Process 
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|  REFLECTIVE  PRACTICE 
|  THE  BIG  IDEA 

Reflective  Practice,  as  a  Learning  Strategy  in  the  YES  Program,  builds  on  the  work  of 
Donald  Schon  (1983).  He  discusses  this  process  as  reflecting  on  action  in  order  to 
engage  in  continuous  learning.  In  the  YES  Program,  reflective  practice  for  teens  involves 
thinking  about  their  work  to  improve  it  and  gain  a  deeper  level  of  understanding. 


|  DESCRIPTION  OF  USE 

Debriefing  after  teaching  inquiry-based  activities  with  younger  children  from 
Community  Partners  or  with  general  public  visitors  in  Saint  Louis  Science  Center 
galleries  provides  frequent  opportunities  for  application  of  the  Reflective  Practice  in  the 
YES  Program. 

For  example,  in  Summertime  Science,  a  group  of  YES  Teens  designed  a  series  of 
activities  to  help  younger  children  from  Community  Partner  organizations  understand 
plants.  They  first  try  out  the  activities  with  one  another.  After  the  tryout,  educators  ask 
them  to  discuss  (1)  what  went  well,  and  (2)  what  did  not  go  well.  After  they  discuss  both 
aspects  of  the  experience,  the  educator  asks  them  what  they  could  change  about  the 
activity  to  make  it  work  better.  Ideas  may  include  giving  clearer  instructions  about  how 
to  participate  or  introducing  concepts  from  botany  using  language  a  5-year-old  can 
easily  understand.  The  teens  repeat  the  activity  with  one  another  several  times  to 
become  comfortable  in  their  roles,  implement  successive  improvements,  and  prepare  to 
offer  the  activity  to  younger  children. 

On  the  first  day,  after  groups  of  children  from  Community  Partner  organizations  have 
left,  the  educators  leading  the  component  gather  teens  together  to  reflect  on  their 
experiences  of  facilitating  the  activity  with  younger  children.  Often,  the  first  day  does 
not  go  smoothly.  Again,  the  educators  ask  teens  to  identify  (1)  what  went  well,  and  (2) 
what  did  not  go  well.  Teens  may  note  that  one  thing  that  went  well  was  the  inquiry 
process  gave  them  the  freedom  not  to  have  to  know  the  right  answer  but  to  work  with 
children  to  find  an  answer.  Then,  as  a  group,  the  teens  will  incorporate  successful 
practices  and  develop  ways  to  make  the  activity  work  better. 

Teens  and  educators  engage  in  this  reflective  practice  for  about  an  hour  each  day.  At  the 
end  of  the  first  week,  educators  ask  teens  to  consider  what  changes  they  have  made 
across  all  the  days  and  to  consider  if  they  can  draw  any  more  general  conclusions  about 
what  did  and  did  not  work  well  with  other  children.  Teens  may  also  share  insights  about 
their  own  deepening  understanding  about  the  structure  of  plants  based  on  comments  or 
questions  from  younger  children.  Other  may  say  that  they  need  to  review  the  science 
concepts  because  areas  they  thought  they  understood  were  vague  and  fuzzy  when  they 
tried  to  explain  them  to  younger  children. 


Klein  Consulting— Version  1.0— October  15, 2013 


51 


A56 


Reflective  Practice  also  is  used  to  debrief  after  inquiry-based  learning  experiences.  For 
example,  in  the  Biofuels  STEM  Component,  groups  of  teens  share  their  progress  with 
other  teams  developing  new  designs  that  save  or  produce  energy.  In  sharing  what 
worked  and  what  did  not,  some  teens  will  talk  about  aspects  of  a  scientific  concept  such 
as  magnetism  or  gravity  that  they  understood  more  deeply  as  the  result  of  their  testing. 
Others  may  observe  they  had  slowed  down  their  own  progress  by  failing  to  keep  records 
of  the  measurements  and  having  to  repeat  tests  using  the  same  equipment  and 
materials.  Still  others  will  discuss  their  findings  about  a  specific  fuel  source  providing 
ideas  for  other  groups  to  try.  This  process  of  presentation  and  reflections  mirrors  the 
discussions  in  research  laboratories. 

|  DELIBERATE  DESIGN 

One  purpose  of  Reflective  Practice  in  the  YES  Program  is  to  provide  a  process  for  teens 
to  make  explicit  connections  between  their  experiences  and  the  meaning  of  their 
experiences.  Without  reflection,  experiences  remain  unexamined  and  lack  meaning.  For 
example,  teens  who  noticed  briefly  that  they  were  able  to  answer  questions  posed  by 
younger  children  might  not  consider  it  deeply,  remember  the  experience,  and  recognize 
that  they  have  mastered  a  science  concept.  Another  purpose  is  to  develop  teens’  capacity 
to  think  about  their  own  learning  (meta-cognition).  This  skill,  scaffolded  by  educators, 
helps  teens  develop  as  independent  learners,  accepting  the  responsibility  of  identifying 
what  they  do  and  do  not  understand  and  what  they  can  and  cannot  do.  Finally,  through 
reflective  practice  successful  practices  are  clearly  identified  and  incorporated,  and 
practices  that  are  not  successful  become  opportunities  to  improve  rather  than  failures. 


|  RESULTS 

Results  of  this  learning  strategy  include  teens: 

•  Increasing  their  capacity  to  learn  from  experience 

•  Increasing  skills  in  directing  their  own  learning 

•  Increasing  their  perception  of  the  meaningfulness  of  their  experiences 

•  Developing  a  better  understanding  of  the  idea  that  mistakes  and  problems  can  be 
learned  from  and  solved 


I  PROGRAM  DOCUMENTS 


YES  Reflection  and  Evaluation  Energy  and  Biofuels  Process 
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|  ROLE  MODELS  AND  MENTORS 

|  THE  BIG  IDEA 

Teens  need  caring  and  successful  older  teens  and  adults  in  their  lives.  Adults  may  serve 
as  role  models  by  opening  up  ideas  about  what  someone  with  whom  they  identify  can 
accomplish,  allowing  them  to  consider  or  reconsider  education  and  career  paths  they 
may  have  written  off.  Caring  adults  and  older  teens  mentor  by  providing  support  and 
advice  as  teens  face  the  challenges  and  big  decisions  of  their  adolescence. 

|  DESCRIPTION  OF  USE 

Through  participation  in  the  YES  Program,  teens  have  a  wide  variety  of  opportunities  to 
identify  positive  role  models.  In  a  STEM  Component,  they  may  be  supervised  by  an 
educator  who  is  an  African  American  engineer  from  in  their  neighborhood  or  a  woman 
whose  previous  job  was  working  in  a  robotics  or  neuroscience  research  lab.  Other 
educators  financed  college  degrees  working  and  developing  skills  in  fast  food  jobs,  an 
experience  some  teens  consider  a  dead  end.  At  networking  events,  teens  meet  members 
of  the  community  who  are  scientists,  engineers,  and  entrepreneurs  that  talk  about  their 
work  and  give  advice  about  education  and  careers.  At  these  events,  teens  may  meet  and 
talk  with  men  and  women  working  in  areas  from  aerospace  to  agriscience  or  a  scientist 
from  the  research  lab  at  a  local  corporation.  At  SLSC  events  such  as  Minority  Science 
Showcase,  teens  talk  with  members  of  the  Society  of  Black  Engineers  and  numerous 
Community  Partner  organizations. 

In  additional,  younger  teens  are  supervised  by  older  YES  Teens  during  summer  STEM 
Components.  These  older  teens  display  the  maturity,  professional  behavior,  and  deeper 
understandings  of  STEM  concepts  that  they  have  developed  over  the  course  of  their 
participation  in  the  program.  YES  Teens  also  serve  as  role  models  to  younger  children 
from  Community  Partner  organizations,  illustrating  that  teenagers  from  children’s  own 
neighborhoods  are  good  at  and  enjoy  activities  connected  to  STEM  topics.  Finally,  YES 
Alumni  are  invited  back  to  events  where  they  share  with  current  YES  Teens  their  stories 
of  college,  work,  and  life. 

YES  Teens  also  find  mentors  in  the  program.  Educators  and  managers  act  as  caring 
adults.  As  part  of  the  Journaling  Learning  Strategy,  Learning  Lab  educators  respond  to 
what  teens  share  about  their  lives  and  often  share  things  about  themselves  and  their 
own  lives  and  education.  Since  the  teens  participate  in  the  program  over  the  course  of 
several  years,  educators  and  managers  watch  them  grow  and  succeed.  Teens  share 
stories  about  success  in  school,  overcoming  challenges  in  their  lives,  and  reaching 
important  goals  such  high  school  graduation  or  college  acceptance.  YES  staff  members 
celebrate  these  important  accomplishments  with  teens  and,  often  knowing  the  obstacles 
they  may  have  encountered,  offer  sincere  congratulations  based  on  long-time 
relationships. 
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|  DELIBERATE  DESIGN 

The  purpose  of  Role  Models  in  the  YES  Program  is  to  expose  teens  to  successful  adults 
with  whom  they  can  identify  in  order  to  expand  their  ideas  about  what  they  can 
accomplish  educationally  and  in  careers.  The  purpose  of  Mentors  in  the  YES  Program  is 
to  connect  teens  to  caring  adults  who  can  guide  them  through  adolescent  challenges  and 
celebrate  their  accomplishments. 


|  RESULTS 

In  evaluation  interviews,  teens  hared  instances  where  an  adult  they  encountered  in  the 
YES  Program  served  as  an  important  role  model,  opening  their  eyes  to  their  own 
potential.  As  one  explained: 

At  first  I  just  thought  scientists  were  like  people  in  lab  coats,  just  working  in  like 
a  lab,  laboratory.  Scientists  can  be  anybody.  [My  YES  supervisor]  don’t  look  like 
a  scientist  but  he  is.  (July  2013  YES  Teen  Interview,  Senior) 

Teens  also  shared  stories  about  how  the  relationship  with  a  caring  adult  made  a 
difference  in  their  lives. 


I  PROGRAM  DOCUMENTS 


YES  Networking  Event  Attendance  List  (names  included) 

YES  Mentor  Job  Descriptions 

YES  Networking  Invite  Letter 

YES  Networking  Event  Flyer 

YES  Networking  Event  Flyer 
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PROGRAM  STRUCTURE 


THE  BIG  IDEA 

The  YES  Program  structure  supports  year-round,  prolonged  engagement  across  four- 
and-a-half  or  more  years  of  an  adolescent’s  life.  Sessions  are  held  five  days  a  week 
during  the  summer  and  on  Saturdays  during  the  school  year.  The  structure  provides 
New  Teens  with  orientation  and  an  initial  work  experience  during  their  first  semester. 
Teens  can  then  pursue  special  interests  through  a  variety  of  STEM  Components 
throughout  the  rest  of  their  five  years.  Some  older  teens  work  as  interns  after  their 
junior  or  senior  year  in  high  school. 


I  DESCRIPTION 
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Elements  of  the  YES  Program  structure  include  (l)  Learning  Labs,  scheduled  units  of 
time  focused  on  specific  learning  objectives,  and  (2)  key  process  through  which  teens 
are  recruited  to  the  program  and  scheduled  into  Learning  Labs. 

There  are  two  major  types  of  Learning  Labs  in  the  YES  Program.  In  Components,  teens 
engage  in  STEM  projects  and  activities.  In  College  Prep,  they  learn  skills  and  accomplish 
tasks  to  make  good  choices  for  their  future.  Figure  9  shows  the  two  types  of  Learning 
Labs. 


Components 
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STEM 
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Figure  9.  Learning  Labs  in  the  YES  Program 

The  New  Teens  Lab  lays  the  foundation  for  the  rest  of  the  YES  experience  as  young 
people  learn  about  the  Science  Center  as  a  workplace,  participate  in  a  variety  of  inquiry- 
based  experiences,  and  begin  developing  skills  that  they  will  expand  through  the  next 
four  years  of  their  YES  experience  to  help  them  succeed  in  college  and  their  careers. 
After  the  New  Teens  Lab  participants  are  ready  for  their  first  paid  work,  and  which  they 
being  in  Summertime  Science,  where  they  teach  science  activities  to  younger  children 
from  Community  Partner  organizations. 

After  their  first  orientation  and  first  summer,  teens  participate  in  STEM  Components. 
These  project-  and  inquiry-based  experiences  are  offered  year  round.  Topics  reflect 
important  issues  and  concerns  in  the  teen’s  world— energy,  food  systems,  and 
technology  development. 

Older  teens  may  work  as  interns.  Some  assist  educators  in  supervising  teens  in 
Summertime  Science  and  summer  STEM  Components.  Others  work  as  summer  interns 
in  Community  Partner  organizations  or  local  businesses.  Interns  use  the  knowledge  and 
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skills  they  developed  in  the  YES  Program  to  serve  as  role  models  and  mentors  to 
younger  teens  or  to  gain  wider  experience  in  the  St.  Louis  community. 

As  part  of  Saturday  school-year  sessions,  teens  attend  College  Prep  Learning  Labs. 
Theses  labs  are  offered  by  grade  level,  providing  the  support  teens  need  to  prepare  for 
life  after  high  school. 

Teens  enter  and  move  through  these  experiences  through  two  key  processes: 
Recruitment  and  Selection.  To  recruit  teens,  YES  staff  members  send  Community 
Partner  organizations  specific  numbers  of  applications.  The  Community  Partners 
identify  young  people  they  think  would  benefit  from  the  program.  After  the  New  Teens 
Lab,  teens  submit  their  preferences  for  STEM  Components  in  which  they  would  like  to 
participate.  Staff  members  place  teens  in  one  of  their  top  three  choices. 

The  YES  Program  operates  year-round.  During  fall  and  spring  semesters,  teens  attend 
Saturday  Learning  Labs.  During  summer,  teens  work  five  days  a  week  in  components 
and  no  College  Prep  is  offered.  During  the  school  year,  different  sets  of  teens  attend 
morning  or  afternoon  sessions.  All  participating  teens  attend  College  Prep  learning  labs 
that ,  which  are  scheduled  between  the  morning  and  afternoon  sessions.  During 
summer,  teens  work  five  days  a  week  in  components.  No  College  Prep  is  offered  during 
this  busy  term,  but  youth  development  activities  are  included  as  part  of  the  components. 
Figure  10  shows  the  year-round  programming. 
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Figure  10.  Yearly  Program  Structure 

Now  let’s  see  how  an  individual  teen  moves  through  the  five-year  program  structure.  In 
Year  l,  teens  enter  the  program  through  Community  Partners.  Recruitment  starts  in  the 
late  fall  and  is  completed  by  early  in  the  new  year.  During  the  spring  semester  of  their 
first  year,  New  Teens  are  not  paid.  They  participate  as  volunteers  in  the  New  Teen 
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Component,  getting  oriented  to  the  program  and  learning  work  skills  and  the  science 
and  activities  they  will  use  in  Summertime  Science.  During  the  summer,  New  Teens 
develop  their  work  skills  by  teaching  children  from  Community  Partner  organizations, 
reflect  on  their  experiences,  and  refine  their  understanding  of  the  underlying  science 
concepts.  At  the  end  of  the  summer,  they  submit  their  preferences  and  are  placed  in  a 
STEM  Component  for  the  fall.  Figure  n  shows  Year  l  program  structure. 


Figure  ll.  Year  l  Program  Structure 

The  program  structure  for  Years  2  through  5  is  similar.  In  the  fall  of  Year  2,  College  Prep 
is  introduced.  Teens  pursue  their  own  interests  by  selecting  and  participating  in  project- 
based  STEM  Components.  College  Prep  continues  in  each  fall  and  spring  in  Years  2 
through  5,  with  learning  objectives  changing  to  meet  the  needs  of  teens  as  college  and 
life  after  high  school  grow  closer.  In  Year  3,  a  few  teens  who  may  have  entered  the 
program  in  later  grade  levels  begin  to  move  into  internships;  interns  may  assist  an 
educator  in  supervising  a  New  Teens  Lab  or  a  STEM  Component.  Internships  may  also 
allow  teens  to  work  in  Community  Partner  Organizations.  A  few  teens  who  entered  the 
YES  Program  as  eighth  graders  may  participate  for  a  fifth  year  of  the  program.  Figure  12 
shows  the  program  structure  in  Years  2  through  5. 
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Figure  12.  Year  2-5  Program  Structure 


DELIBERATE  DESIGN 

The  design  of  the  YES  Program  structure  is  based  on  the  following  rationale: 

•  As  teens  enter  the  program,  they  need  the  opportunity  to  get  excited  about  STEM 
topics,  develop  inquiry  and  workplace  skills  that  may  be  unfamiliar,  and 
understand  the  culture  of  the  Saint  Louis  Science  Center.  The  New  Teens  Labs 
and  Summertime  Science  provide  this  orientation. 

•  The  selection  of  STEM  Components  allows  teens  to  pursue  and  build  deeper 
knowledge  in  specific  areas  of  interest.  Not  all  teens  have  the  same  interests,  and 
the  aim  of  the  YES  Program  is  to  get  teens  deeply  engaged  in  STEM. 

•  The  four-  to  five-year  program  length  provides  the  basis  for  in-depth  learning 
and  the  development  of  strong,  caring  relationships  that  contribute  to  teens 
accomplishing  the  program  impacts. 

•  Year-round  sessions  keep  teens  involved  in  the  program  and  sustain 
communication  and  relationships.  Because  underserved  teens  move  frequently 
and  may  change  high  schools  several  times,  the  YES  Program  provides  a 
consistent  safe  environment  throughout  the  year. 

•  Multi-grade-level  participation  in  STEM  Components  allows  for  older  teens  to 
assume  leadership  roles  in  project-based  learning  and  act  as  role  models  to 
younger  teens  in  the  development  of  workplace  skills. 

•  More  specific  grade-level  groupings  in  College  Prep  responds  to  the  differing 
needs  of  teens  as  college  and  life  after  high  school  grow  closer. 
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•  Internships  allow  teens  to  take  on  greater  levels  of  responsibility  and  implement 
more  fully  the  workplace  and  leadership  skills  they  have  developed  in  the 
program. 


RESULTS 

A  consistent,  yearly  structure  makes  planning  and  design  of  learning  experiences  more 
effective  and  efficient.  Materials,  technology,  space,  plans,  and  procedures  can  be 
developed,  implemented,  and  refined  without  the  constant  need  for  reinvention.  This 
leaves  time  for  real  improvement  and  innovation.  Finally,  a  clearly  structured  program 
allows  teens  to  understand  what  they  are  doing  and  where  they  are  going  over  the 
course  of  a  five-year  experience. 

NEW  TEENS  LAB 


THE  BIG  IDEA 

The  New  Teens  Lab,  held  in  the  spring  of  each  year,  builds  a  foundation  for  the  rest  of 
YES  Teens’  four-  to  five-year  experience.  New  Teens  is  the  term  used  to  refer  to  the 
entering  cohort  of  young  people  who  enter  the  program  in  the  spring  of  each  year. 


DESCRIPTION  OF  USE 

In  spring  of  each  year,  New  Teens  attend  six  to  eight  Saturday  sessions.  While  some 
activities  may  vary,  the  New  Teens  Lab  generally  follows  the  curriculum  presented  in  the 
YES  2012  New  Teens  Lab  Curriculum.  Each  session,  teens  arrive  to  find  an  agenda  of 
the  day’s  activities  written  on  a  white  board  or  posted  on  the  door.  The  agenda  follows  a 
consistent  pattern  similar  to  the  one  they  will  encounter  in  the  rest  of  the  program: 

Agenda 

10:00  a.m.  -10:15  a.m.  Town  Hall 
10:15  a.m. -10:30  a.m.  Journaling 
10:15  a.m. -10:45  a.m.  Icebreaker 
10:45  a.m.-n:i5  a.m.  21st  Century  Skill 
11:15  a.m. -12:45  P-m-  Break 
I2:45p.m.-i:45  p.m.  Science  Activity 
1:45  p.m.-2:oo  p.m.  Reflection,  Clean  up 

A  Town  Hall  meeting  begins  the  day,  allowing  time  for  announcements  and  questions. 
Then,  with  music  playing  the  background  to  build  a  reflective  atmosphere,  teens  write  in 
their  journals  about  what  is  going  on  in  their  lives.  (After  each  session,  educators  will 
read  each  journal  entry  and  respond.)  This  is  followed  by  an  Icebreaker  (games  often 
involving  physical  activity),  a  teen  favorite  providing  ways  for  teens  get  to  know  one 
another,  engage  in  a  physical  activity,  and  have  fun. 

The  next  activity  focuses  on  the  development  of  specific  21st  Century  Skills.  For 
example,  in  one  lab  teens  participate  in  a  communication  activity  about  body  language 
that  builds  an  understanding  that  people  communicate  even  when  they  are  not  talking. 
After  a  break,  a  STEM  activity  is  next.  In  the  New  Teens  Lab,  STEM  activities  build  the 
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idea  that  science  is  fun  and  to  introduce  foundational  concepts  about  the  science  process 
as  well  as  several  big  ideas  in  chemistry,  physics,  and  biology.  The  day  ends  with 
reflection,  allowing  teens  to  make  explicit  connections  about  the  ideas  and  concepts  they 
have  been  introduced  to  through  activities. 


DELIBERATE  DESIGN 

As  an  essential  element  of  the  overall  program  design,  the  New  Teens  Lab  recognizes 
that  teens  need  support  to  successfully  navigate  a  new  learning  environment  that  may 
be  different  from  others  they  have  encountered.  Teens  are  introduced  to  the  SLSC  as  a 
workplace  and  to  21st  Century  Skills  and  STEM  Concepts.  They  become  familiar  as  well 
with  the  YES  session  format. 


RESULTS 

After  participating  in  the  New  Teens  Lab  members  of  the  entering  cohort  grow  and 
change  in  several  ways.  They: 

•  Demonstrate  an  increased  comfort  level  with  learning  and  teaching  STEM 
concepts. 

•  Act  as  YES  team  members. 

•  Articulate  YES  goals,  policies,  and  procedures. 

•  Articulate  broad  concepts  in  science. 

Members  of  the  YES  staff  have  carefully  identified  some  of  the  foundational  concepts  in 
science  based  on  national  standards  and  other  resources  prescribing  the  knowledge  U.S 
citizens  and  workers  need.  As  part  of  the  New  Teens  Lab,  they  become  familiar  with  the 
language  and  phenomena  underlying  these  big  ideas  of  science: 

•  Science  is  a  human  endeavor. 

•  Science  is  a  process. 

•  Science  involves  the  study  of  both  living  and  non-living  things. 

•  Matter  exists  in  various  states. 

•  There  are  several  types  of  energy. 


PROGRAM  DOCUMENTS 


YES  New  Teens  Lab  Curriculum  2012 
YES  Scientific  Process  Activity  2012 


REFERENCES 
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SUMMERTIME  SCIENCE 


THE  BIG  IDEA 

Each  summer,  YES  Teens  teach  STEM  concepts  to  younger  children  from  Community 
Partners  and  Science  Center  visitors  through  inquiry-based  activities  offered  onsite  in 
the  Taylor  Community  Science  Resource  Center  or  in  the  SLSC  galleries.  They  also 
develop  workplace  skills  and  carry  out  design-  or  inquiry-based  projects.  Prior  to  2013, 
Summertime  Science  Components  focused  on  inquiry-based  activities  in  different  STEM 
areas;  these  changed  from  summer  to  summer.  In  summer  2013,  YES  Teens  in  the  new 
cohort  participated  together  in  Summertime  Science,  with  experienced  teens  assisting 
educators  as  supervisors. 


DESCRIPTION  OF  USE 

In  Summertime  Science,  about  50  teens  began  their  day  at  9  a.m.  in  a  classroom  of  the 
Taylor  Center,  signing  into  their  component  and  journaling  about  what  was  going  on  in 
their  lives  outside  the  program.  At  9:30,  they  moved  to  another  classroom  to  prepare  for 
the  first  group  of  children  from  a  Community  Partner  organization.  At  10  a.m.,  working 
as  a  group,  YES  Teens  led  inquiry-based  activities  focused  on  chemistry.  At  11  a.m., 
another  group  arrived,  and  teens  repeated  the  activity,  while  the  first  group  of  children 
participated  in  activities  led  by  another  group  of  YES  teens.  After  a  break  for  lunch, 
teens,  educators,  and  interns  gathered  to  reflect  about  their  experiences,  considering 
what  went  well  and  what  could  have  gone  better,  and  developing  strategies  to  improve 
the  presentation  and  adjust  the  activity  and  materials. 


DELIBERATE  DESIGN 

The  design  and  activities  that  are  part  of  Summertime  Science  serve  several  important 
purposes: 

•  By  teaching,  teens  engage  in  meaningful  work  so  that  learning  new  concepts  and 
skills  become  relevant. 

•  Teaching  STEM  inquiry-based  activities  to  younger  children  allows  some  teens  to 
learn  STEM  concepts  they  may  have  missed  in  earlier  grades  and  build  a  firmer 
foundation  for  high  school  science  subjects. 

•  Designing  and  facilitating  activities  for  younger  children  requires  teamwork  and 
develops  several  21st  Century  Skills. 

•  Summertime  Science  provides  a  useful  service  to  Community  Partners  who  bring 
younger  children  to  the  SLSC  as  part  of  their  summer  programs. 

•  Teens  serve  as  role  models  for  younger  children. 

•  Older  YES  Teens  mentor  younger  YES  Teens. 

•  Summertime  Science  provides  summer  jobs  for  teens  and  allows  them  to  practice 
and  develop  workplace  skills. 


RESULTS 

In  Summertime  Science,  teen  learn  to  follow  workplace  procedures  such  as  clocking  in 
and  out,  wearing  uniforms,  and  taking  responsibility  for  their  own  work  performance. 
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New  Teens  develop  skills  in  communication  through  presentation  and  working  with 
others  in  teams.  By  teaching  younger  children,  they  gain  deeper  understanding  of  STEM 
concepts.  Interns  (referred  to  as  Junior  Interns  in  this  Component)  develop  leadership 
skills  and  practice  communication  by  mentoring  younger  teens,  answering  questions, 
and  solving  problems.  By  designing  and  refining  inquiry-based  learning  experiences  for 
younger  children  or  Science  Center  visitors,  teens  develop  expertise  in  using  scientific 
materials  and  equipment.  All  YES  Teens  involved  develop  meta-cognitive  skills  by 
reflecting  on  their  practice.  Supervisors  work  with  teens  to  take  responsibility  for  the 
quality  of  their  own  work  and  learning  through  the  evaluation  of  teens’  work  and 
performance  in  the  following  areas: 

•  Science  Content 

•  Flexibility 

•  Internal/External  Customer  Service 

•  Initiative/Motivation 

•  Punctuality 

•  Team  Player 

•  Communication 

•  Professional  Appearance 

•  Dependability 

•  Quality  of  Work 


PROGRAM  DOCUMENTS 


YES  Teen  Evaluation  Summer  2013 
YES  Teen  Intern  Evaluation  Summer  2013 


REFERENCES  AND  RESOURCES 


Saint  Louis  Science  Center.  (2013).  Elementary  &  Middle  School  Demo:  Density  Tower.  Mystery 
of  Matter  Outreach:  Chemistry  Beyond  the  Classroom.  Assessed  on  October  10,  2013 
from  http://mysteryofmatteroutreach.org/. 

Saint  Louis  Science  Center.  (2013).  Elementary  Demo:  Making  Slime.  Mystery  of  Matter 
Outreach:  Chemistry  Beyond  the  Classroom.  Accessed  on  October  10,  2013,  from 
http:  /  /  mysteryofmatteroutreach.org/. 
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STEM  COMPONENTS 


THE  BIG  IDEA 

STEM  Components  focus  on  one  specific  area  STEM  area,  allowing  teens  to  pursue  their 
own  interests  and  engage  in  meaningful  work  through  teaching  and  project -based 
learning.  STEM  Components  focus  on  community-based  problems  relevant  to  the  YES 
Teens’  current  and  future  lives.  Currently,  these  specialized  components  are  offered  year 
round,  on  Saturdays  during  the  school  year,  and  as  part  of  the  YES  summer  program. 
School  year  components  prepare  teens  for  summer  work,  providing  the  opportunity  for 
extended  inquiry  and  project -based  learning. 


DESCRIPTION  OF  USE 

In  summer  2013,  about  15  returning  teens  participated  in  each  of  these  six  STEM 
Components: 

•  Agriscience 

•  Exhibit  Lab 

•  Design  Engineering 

•  Mystery  of  Matter 

•  Astronomy 

•  Science  of  Learning 

Teens  presented  inquiry-based  learning  experiences  to  Science  Center  visitors  in  the 
Mystery  of  Matter  and  Astronomy  Components.  In  the  Agriscience  Component,  teens 
grew  plants  and  studied  food  systems,  and  some  planned  projects  to  sell  fresh  produce 
to  local  markets.  In  the  Exhibit  Lab,  teens  designed  and  prototyped  exhibits,  testing 
their  designs  with  SLSC  visitors.  The  Design  Engineer  teens  collaborated  with  HOSCO 
Foods  interns  to  design  and  build  aquaponic  systems.  Science  of  Learning  participants 
designed  learning  experiments  involving  laboratory  rats  and  built  mazes  to  run  their 
experiments. 


DELIBERATE  DESIGN 

Each  STEM  Component  features  the  following  elements  of  Deliberate  Design: 

•  Focus  on  issues  relevant  to  teens’  lives  so  they  can  learn  in  context  through 
meaningful  work 

•  Use  of  standard  YES  Program  elements,  including  Word-of-the  Day,  Quote-of- 
the-Day,  Icebreakers,  and  Journaling. 

•  21st  Century  Skills  and  the  7  Developmental  Needs  of  Teens. 


RESULTS 

At  the  end-of-summer  event,  teens  shared  their  work  with  parents,  other  family 
members,  teens  in  other  components,  and  YES  alumni.  Teens  in  the  Engineering  Design 
Component  showed  their  parents  and  family  members  the  aquaponic  systems  they  had 
designed  and  built  in  a  competition  with  another  component.  Parents  also  had  the 
opportunity  to  see  exhibit  prototypes  for  a  periscope  exhibit  designed  by  teens  in  the 
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Exhibit  Lab  Component.  Agriscience  Teens  planned  the  menu  and  prepared  a  buffet 
spread  that,  featured  healthful  foods  they  had  learned  to  cook,  ranging  from  watermelon 
salad  to  Brussels  sprout  canapes,  and  prepared  with  food  they  had  grown  on  Science 
Corner,  a  garden  area  adjacent  to  the  Taylor  Center.  Family  members  watched  rats  run 
mazes  as  teens  described  the  experiments  they  had  designed  and  the  conclusions  they 
had  reached.  Family  members  also  participated  in  samples  of  the  hands-on  inquiry 
based  activities  teens  had  led  that  summer  as  part  of  the  Mystery  of  Matter  and 
Astronomy  Components. 


PROGRAM  DOCUMENTS 


2013  YES  Summer  Component  Overview 
YES  Agriscience  Curriculum-Summer  2013 
YES  Component  Descriptions  Summer  2012 
YES  Energy  and  Biofuels  Curriculum-Summer  2012 
YES  Mystery  of  Matter  Curriculum-Fall  2012 


TEEN  INTERNSHIPS 


THE  BIG  IDEA 

YES  Teens  participate  as  interns  onsite  as  part  of  Summertime  Science  and  STEM 
Components  as  well  as  testing  and  developing  their  skills  and  knowledge  offsite  in 
summer  internships  at  Community  Partner  locations  around  St.  Louis. 


DESCRIPTION  OF  USE 

In  Summertime  Science,  10  Junior  Interns  assisted  educators  in  supervising  New  Teens 
in  their  first  work  experience.  In  this  component,  Junior  Interns  worked  with  New 
Teens  as  they  designed  and  adapted  inquiry-based  activities  form  the  Mystery  of  Matter 
Curriculum.  Using  their  own  experience  as  a  background,  they  worked  with  educators  to 
monitor  New  Teens’  understanding  of  the  underlying  science  concepts  related  to  states 
of  matter  and  density.  As  New  Teens  taught  activities  to  one  another,  Junior  Interns 
participated  in  reflective  practice  to  help  them  further  refine  their  presentation  skills 
while  handling  the  materials  and  equipment  required.  Junior  Interns  sometimes  worked 
with  small  groups  of  teens,  helping  them  solve  specific  task-related  and  teamwork 
issues.  After  two  weeks  of  design  and  practice,  children  from  Community  Partner 
organizations  arrived  at  the  Taylor  Center  of  the  SLSC  to  participate  in  the  activities. 
Junior  Interns  monitored  groups  of  teens  during  sessions  as  educators  rotated  among 
multiple  groups. 

Meanwhile,  53  YES  Teens  participated  in  internships  at  Community  Partner  sites 
through  the  YES  in  the  Community  Component.  First,  teens  participated  in  a  Work 
Readiness  Program  designed  to  summarize  the  knowledge  and  skills  they  had  developed 
in  the  YES  Program  and  translate  it  into  the  new  environments  in  which  they  would  be 
working. 
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DELIBERATE  DESIGN 

Internships  provide  opportunities  for  YES  Teens  to  take  on  work  that  involves  greater 
levels  of  responsibility  as  they  mature  and  move  through  the  program. 


RESULTS 

YES  staff  members  and  Community  Partners  reported  that  teens  in  internship  roles 
displayed  leadership  and  professionalism.  These  are  key  workplace  skills  developed  in 
the  program.  In  the  Summertime  Science  STEM  Component,  one  educator  observed  the 
commitment  and  skill  with  which  Junior  Interns  approached  their  work.  He  observed 
Junior  Interns  correcting  behavior  on  a  one-to-one  basis  and  developing  relationships 
and  teamwork  among  the  groups. 


PROGRAM  DOCUMENTS 


YES  in  the  Community  Internships  Summer  2013 

YES  Community  Partner  Letter  of  Teen  Placement 

YES  Work  Readiness  Curriculum  April-May  2013 

YES  Summer  Internship  Placement  Letter-General  Teen  Copy 


COLLEGE  PREP 


THE  BIG  IDEA 

College  Prep  learning  labs  are  offered  between  morning  and  afternoon  Saturday 
sessions.  Teens  are  assigned  to  three  grade-level  learning  labs:  Freshman  and 
sophomores  are  grouped  together  and  junior  and  seniors  each  participate  in  specialized 
sessions. 


DESCRIPTION  OF  USE 

College  Prep  learning  labs  focus  on  the  specific  needs  of  teens  as  life  after  high  school 
draws  nearer.  In  the  Freshman-Sophomore  College  Prep  learning  labs,  teens  participate 
in  large  group  experiences  and  explore  interests.  During  the  junior  and  senior  years, 
College  Prep  features  more  individualized  experiences  as  teens  write  resumes,  identify 
colleges,  and  plan  how  to  fund  their  college  experience.  YES  staff  members  developed  a 
year-by-year  checklist  for  teens  to  use  in  preparing  for  college.  Activities  and 
individualized  work  with  teens  by  educators  in  College  Prep  learning  labs  support  teens 
in  taking  responsibility  for  their  own  preparations  for  college  and  life  after  high  school. 

Freshman  Year  Checklist 

•  Review  High  School  College  Prep  Guidelines 

•  Enroll  in  a  college  preparatory  curriculum 

•  Start  thinking  about  reasons  for  attending  college 

•  Become  familiar  with  college  entrance  requirements 
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•  Participate  in  extracurricular  activities 

•  Attend  summer  camp  at  a  college  to  experience  a  college-like  atmosphere 

•  Start  a  bank  account 

•  Continue/start  saving  for  college 

•  Meet  with  your  high  school  counselor  and  let  them  get  to  know  you  and  your 
goals,  career  aspirations,  schools  you  are  considering,  etc. 

•  Explore  careers  on  the  Internet 

•  Study  hard  and  do  well  in  school:  From  now  on  your  grades  will  count  toward 
college  and  will  show  up  on  your  permanent  record 

•  Learn  keyboarding  and  computer  skills 

•  Identify  tutors  and  other  resources  for  class  subjects,  if  needed 

Sophomore  Year 

•  Continue  to  take  and  plan  challenging  high  school  courses 

•  Continue  to  meet  with  your  college/career  counselor  at  least  once  a  semester 

•  Keep  exploring  careers  on  the  Internet 

•  Think  about  what  kind  of  education/training  different  careers  require 

•  Consider  your  reasons  for  going  to  college  and  how  they  relate  to  your  career 
interests 

•  Participate  in  extracurricular  activities 

•  Take  the  PLAN  Test  and  take  the  PSAT  Test 

•  Pick  up  an  ACT  Prep  book 

•  Get  a  part-time  job  outside  of  YES  Program  (if  desired  and  possible) 

•  Continue/start  saving  for  college 

•  Get  to  know  who  you  are  as  a  person  and  as  a  learner 

Junior  Year 

Continue  to  take  and  plan  challenging  high  school  courses 

•  Update  High  School  College  Prep  Worksheet 

•  Meet  with  your  college/ career  counselor  at  least  once  a  semester 

•  Keep  exploring  careers  on  the  Internet 

•  Volunteer  for  activities  and  clubs  related  to  career  interests 

•  Participate  in  extracurricular  activities,  academic  clubs,  and  volunteer 

•  Keep  your  grades  up 

•  Pick  up  an  ACT  Prep  book  and  complete  a  practice  test 

•  Register  for  the  ACT.  You  should  take  it  by  this  spring.  If  not,  take  it  early  in  your 
senior  year  or  in  the  summer  before  classes  start. 

•  Read  key  information  about  the  ACT  test  and  prepare  for  the  test 

•  Talk  with  your  parents  and  high  school  counselor  about  colleges  that  interest  you 

•  Attend  a  college  fair 

-  Collect  information  from  five  colleges 

•  Prepare  a  list  of  five  to  seven  questions  to  ask  on  campus  visits 

•  Visit  one  college  that  interests  you 

•  Create  an  academic  resume  of  your  high  school  experience 
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•  Consider  putting  together  a  portfolio  that  highlights  your  special  skills  and 
talents 

•  Investigate  scholarship  opportunities 

•  Get  a  part-time  job,  apprenticeship,  or  internship;  or  job  shadow  in  a  profession 
that  interests  you 

•  Create  a  professional  email  address  for  correspondence  with  colleges 
(HotChica@aol.com  does  NOT  look  good  on  applications) 

Senior  Year 

Summer  before  Senior  Year 

•  Sign  up  for  the  ACT  (if  you  didn’t  take  it  as  a  junior,  if  you  aren’t  satisfied  with 
your  score,  or  if  you’ve  learned  a  lot  since  you  took  it) 

•  Review  ACT  test  results  and  retest  if  necessary 

•  Review  application  requirements  for  colleges  that  interest  you 

-  Pick  colleges  you  will  apply  to.  You  need  to  select  at  least  one  safety  school, 
one  reach  school  and  one  target  school. 

-  Write  your  college  essay  to  each  college  you  are  applying  to.  Have  them 
reviewed  by  a  person  you  trust  (i.e.,  YES  Supervisor). 

August-December 

•  Visit  with  your  school  counselor  to  make  sure  you  are  on  track  to  graduate  and 
fulfill  college  admission  requirements 

•  Keep  working  hard  all  year;  second  semester  grades  can  affect  scholarship 
eligibility 

•  Participate  in  extracurricular  activities,  academic  clubs,  and  volunteer 

•  Consider  taking  an  Advanced  Placement  (AP)  class  or  test 

•  Ask  for  personal  references  from  teachers,  school  counselors,  or  employers  early 
in  the  year  or  at  least  two  weeks  before  application  deadlines.  Follow  your 
school's  procedure  for  requesting  recommendations. 

•  Visit  with  admissions  counselors  who  come  to  your  high  school 

•  Attend  a  college  fair.  Collect  information  from  five  colleges. 

•  Review  worksheet  on  common  college  application  mistakes 

•  Apply  for  admission  at  the  colleges  you’ve  chosen 

•  See  your  school  counselor  for  help  finding  financial  aid  and  scholarships 

•  Find  out  if  you  qualify  for  scholarships  at  each  college  you  have  applied  to 

•  Start  the  financial  aid  application  process;  watch  for  Deadlines 


DELIBERATE  DESIGN 

YES  Teens,  even  those  doing  well  in  school  coursework,  have  little  individualized 
support  at  home  or  at  school  in  undertaking  many  of  the  tasks  required  for  college 
admission.  YES  educators  share  their  own  experiences,  and  teens  work  together  to 
complete  move  through  the  steps  for  college  application. 
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RESULTS 

Evaluators  found  that  YES  Teens  who  remain  in  the  program  until  their  senior  year  in 
high  school  graduate  at  a  higher  rate  than  their  peers  in  the  same  area  schools,  with 
96.36%  of  YES  Teens  that  could  be  contacted  graduating  in  2012  and  100%  in  2013. 
From  the  graduating  class  of  2012,  55%  of  YES  Teens  planned  to  attend  a  four -year 
college  or  university,  and  46%  in  2013.  Still,  College  Prep  appears  plays  a  strong  role  in 
these  results.  One  of  the  YES  educators  described  the  value  she  saw  in  working  with 
high  school  juniors  and  seniors  in  the  following  way. 

I  think  the  most  valuable  thing  is  probably  just  being  able  to  have— I  work  with 
older  teens ,  and  so  we  have  those  conversations  about  college  and  where  they’re 
going  and  be  sure  that  they  have  everything  done  that  they  need  to  have  done 
and  that  they’re  as  prepared  as  possible.  And  I  really  think  I  was  in  a  position  to 
encourage  them.  (August  2012  YES  Summer  Intern  Focus  Group) 


PROGRAM  DOCUMENTS 


YES  College  Prep  Checklist-2013 

YES  College  Prep  2013-2014  (Youth  Development  Activities) 
YES  College  Prep  FALL  2012  (Academic  Activities) 
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OUTPUTS  AND  IMPACTS 


OUTPUTS 


Program  records  provide  a  source  of  valuable  information 
about  program  outputs.  Outputs  tracked  in  the  YES 
Program  focus  on  the  number  of  teens  served,  learning 
labs  offered,  and  attendance.  Information  about  outputs 
informs  several  stakeholding  groups.  For  managers, 
examining  outputs  help  them  make  decisions  about  the 
types  and  range  of  program  offerings  as  well  as  the 
decisions  about  the  level  and  assignment  of  staff. 

Attendance  may  highlight  program  issues.  For  the  SLSC 
administrators,  the  level  of  outputs  informs  decisions 
about  budget  and  funding.  For  funders  and  Community 
Partners,  the  level  of  outputs  informs  their  judgments  about  whether  the  program 
reaches  too  many  or  too  few  teens  to  be  a  valuable  investment  or  to  provide  important 
adequate  community  benefit. 


YES  OUTPUTS 

In  the  YES  Program,  knowing  the  overall  numbers  of  participating  teens  by  grade  level 
supports  decisions  about  the  number  of  learning  labs  that  need  to  be  scheduled  and  how 
to  distribute  staffing  to  lead  these  sessions.  Teens  attend  grade-level-specific  sessions  of 
College  Prep.  In  addition,  grade  levels  provide  information  about  the  levels  of  expected 
maturity  and  understanding  among  teens  as  a  whole  and  among  those  assigned  to 
individual  components.  Younger  teens  may  need  more  supervision  in  work  situations. 

Table  6  shows  the  large  number  of  Middle  School  and  freshmen  in  the  YES  Program  in 
2010-2011.  Funding  provided  by  the  Community  STEM  Outreach  project  allowed 
program  expansion  and  a  large  cohort  of  New  Teens  entered  the  program.  This  required 
hiring  additional  staff  members. 

Table  6.  Grade  Levels  of  YES  Teens  by  Year  of  Project 


Outputs  include 
the  number  of 
people  served  and 
the  number  and 
types  of  program 
offerings. 


Grade  Level 

2010-2011 

2011-2012 

2012-2013 

Middle  School 

62 

36 

31 

Freshman 

73 

79 

46 

Sophomore 

50 

79 

68 

Junior 

59 

45 

72 

Senior 

56 

51 

39 

Total  (N) 

300 

290 

256 

The  YES  Program  also  tracks  teens  by  cohort— that  is,  the  year  they  entered  the 
program.  While  most  teens  join  the  program  in  the  spring  of  their  freshman  and 
sophomore  year,  others  join  later.  Figure  13  shows  the  number  of  participating  teens 
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across  a  three-year  time  period  by  cohort.  Tracking  teens  by  cohort  helps  program  staff 
make  decisions  about  the  organization  of  New  Teens  Labs  and  to  anticipate  the  number 
of  internships  that  may  be  appropriate  to  offer. 


Figure  13.  Number  of  Participating  YES  Teens  by  Cohort 

Table  7  shows  of  STEM  Components  by  semester,  with  the  number  of  teens 
participating  by  component.  (Participating  teens  are  defined  as  those  who  attended 
more  than  one  day  in  that  semester.  Teens  who  teen  may  sign  up  for  a  semester  and 
never  attend  are  not  counted  as  participating  in  the  program  that  semester.)  Note  that 
component  names  change  from  semester  to  semester.  While  information  in  this  table 
may  be  helpful  in  tracking  which  components  were  offered  during  what  semester,  it 
does  not  help  program  managers  monitor  the  program  goal  of  offering  students 
experiences  across  different  STEM  disciplines.  For  many  components,  it  is  difficult  for 
stakeholders  such  as  parents,  funders,  and  institutional  administrators  to  identify  the 
STEM  topics  by  component  name. 
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Table  7.  Components  Participation  by  Semester  with  Teen  Participants 


Spring 

2011 

Summer 

2011 

Fall 

2011 

Spring 

2012 

Summer 

2012 

—  CM 

£  O 

CM 

Spring 

2013 

Summer 

2013 

School  Year  Component  Groups 

New  Teens 

105 

- 

- 

61 

- 

- 

48 

- 

Agriscience 

- 

21 

34 

35 

24 

- 

- 

14 

Astronomy 

- 

- 

28 

34 

23 

- 

- 

11 

Atmospheric  Science 

- 

- 

- 

- 

- 

29 

25 

- 

Biofuels  &  Energy 

- 

- 

22 

35 

28 

35 

23 

- 

C3  (Climate  Change) 

14 

- 

- 

- 

- 

- 

- 

- 

Design  IT 

18 

- 

- 

- 

- 

32 

28 

10 

Health 

10 

- 

- 

- 

- 

- 

- 

- 

HOSCO 

- 

- 

- 

- 

- 

- 

- 

16 

Learning  Places 

21 

- 

- 

- 

- 

- 

- 

- 

Neuroscience 

- 

- 

24 

3 

- 

31 

35 

- 

Plant  Biochemistry 

- 

- 

25 

- 

- 

- 

- 

- 

Robotics 

18 

- 

38 

41 

22 

- 

- 

- 

SciJourn 

8 

13 

6 

8 

11 

- 

- 

- 

Science  Corner 

25 

- 

- 

- 

- 

- 

- 

- 

Sea  Perch 

18 

- 

- 

- 

- 

- 

- 

- 

Mystery  of  Matter 

- 

9 

- 

- 

8 

32 

31 

16 

Summer  Component  Groups 

Summertime  Science 

- 

96 

- 

- 

60 

- 

- 

60 

Exhibit  Lab 

- 

- 

- 

- 

71 

- 

16 

10 

Main  Building 

- 

67 

- 

- 

- 

- 

- 

- 

Offsite 

- 

60 

- 

- 

5 

- 

- 

53 

Science  on  the  Go 

- 

10 

- 

- 

- 

- 

- 

- 

Science  of  Learning 

- 

- 

- 

- 

- 

- 

- 

15 

Total 

237 

276 

177 

217 

252 

159 

206 

205 
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To  allow  stakeholders  to  assess  the  distribution  of  offerings  and  participation  by  STEM 
areas,  the  evaluator  merged  numbers  for  components  into  some  broader  disciplinary 
categories.  Table  8  shows  participation  across  semester  by  broader  STEM  areas. 

Table  8.  STEM  Content  Offered  by  Semester  with  Teen  Participants 


Spring 

2011 

Summer 

2011 

Fall 

2011 

Spring 

2012 

Summer 

2012 

Fall 

2012 

Spring 

2013 

Summer 

2013 

New  Teens 

105 

- 

- 

61 

- 

- 

48 

- 

Astronomy 

- 

- 

28 

34 

23 

- 

- 

11 

Biology/Env  Science 

49 

21 

105 

73 

52 

95 

83 

30 

Chemistry 

- 

9 

- 

- 

8 

32 

31 

16 

Engineering  &  Design 

75 

- 

38 

41 

93 

32 

44 

20 

Journalism 

8 

13 

6 

8 

11 

- 

- 

- 

Teaching  Science 

- 

233 

- 

- 

65 

- 

- 

128 

Total 

237 

276 

177 

217 

252 

159 

206 

205 

Youth  programs  need  to  monitor  attendance.  In  the  YES  Program,  attendance  policy 
remains  flexible  to  allow  teens  to  participate  in  sports  and  other  extracurricular  school 
activities,  as  well  as  to  handle  family  issues.  Yet  ongoing  absences  indicate  there  may  be 
a  problem  with  either  the  teen  or  the  program.  For  the  program,  comparing  attendance 
levels  over  time  provides  a  way  to  identify  issues. 

Table  9  provides  an  overview  of  YES  attendance  since  the  Community  STEM  Outreach 
Project  began  in  Fall  2011.  Row  1  shows  the  number  of  Learning  Lab  opportunities 
(number  of  day  that  sessions  were  held  in  each  semester).  Row  2  shows  the  total 
number  of  participating  teens  in  each  of  these  semesters,  and  Row  3  shows  the 
percentage  of  attendance  for  each  semester.  Attendance  percentage  was  calculated  as 
the  number  of  sessions  attended  by  teens— i.e.,  semester  cumulative  total  of 
attendance.). 

Table  9.  Attendance:  Learning  Lab  Opportunities  and  Percent  Attendance 
(Spring  2010— Summer  2013) 


Semester 

Spring 

2011 

Summer 

2011 

Fall 

2011 

Spring 

2012 

Summer 

2012 

Fall 

2012 

Spring 

2013 

Summer 

2013 

Days 

15 

36 

11 

14 

32 

11 

17 

19 

Participating  Teens 

237 

278 

177 

217 

252 

159 

206 

205 

Percent  Attendance 

67.9% 

82.4% 

71.1% 

52.6% 

83.4% 

70.8% 

60.4% 

78.6% 
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IMPACTS 

For  the  YES  Program,  long-term  impacts  include 
college  graduation  rates,  career  choices,  and 
productive  working  lives  among  program  alumni. 

Keeping  track  of  participants  over  long  periods  of  time 
is  very  difficult.  Yet  assessing  the  short-term  impacts 
of  the  YES  Program  during  program  participation  and 
as  participants  exit  can  provide  information  about  the 
effectiveness  of  the  program. 

Evaluators  worked  with  YES  staff  members  to  develop 
a  set  of  impacts  for  the  YES  Program.  We  used 
categories  described  Framework  for  Evaluation  Impacts  of  Informal  Science  Education 
(Friedman,  2008,  p.  21).  Table  10  shows  Informal  Science  Education  (ISE)  impact 
categories  and  definitions.  These  statements  include  both  short-term  and  long-term 
impacts.  Using  a  consistent  set  of  impact  categories  with  definitions  supports  the 
development  of  impact  statements  that  are  useful  and  understandable  across  staff  and 
program  changes.  Funders  and  collaborators  also  will  find  standardized,  clearly 
developed  impact  statements  useful.  Most  importantly,  staff  members  and  other 
stakeholders  can  clearly  understand  what  the  program  aims  to  accomplish  and  organize 
experiences,  activities,  and  resources  to  accomplish  them. 


Impacts  describe  the 
short-  and  long-term 
changes  in  knowledge, 
attitudes,  and  behavior 
among  members  of  the 
target  audience. 
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Table  10.  Impact  Categories  As  They  Relate  to  Public  Audiences  (Friedman,  2008, 
p.  21) 


Impact  category 

Genet  ic  definition 

Awareness,  knowledge  or 
understanding 

Measurable  demonstration  of  assessment  of,  change  m,  or  exercise  of 
awareness,  knowledge,  understanding  of  a  particular  scientific  topic,  concept, 
phenomena,  theory,  or  careers  central  to  the  project 

Engagement  or  interest 

Measurable  demonstration  of  assessment  of,  change  in,  or  exercise  of 
engagement/interest  m  a  particular  scientific  topic,  concept,  phenomena, 
theory,,  or  careers  central  to  the  project 

Attitude 

Measurable  demonstration  of  assessment  of,  change  in,  or  exercise  of  attitude 
toward  a  particular  scientific  topic,  concept,  phenomena,,  theory,  or  careers 
central  to  the  project  or  one’s  capabilities  relative  to  these  areas.  Although 
similar  to  awareness-mterest/engagement,  attitudes  refer  to  changes  in 
relatively  stable,  more  intractable  constructs  such  as  empathy  for  animals  and 
their  habitats,  appreciation  for  the  role  of  scientists  in  society  or  attitudes 
toward  stem,  cell  research 

Behavior 

Measurable  demonstration  of  assessment  of,  change  in,  or  exercise  of 
behavior  related  to  a  STEM  topic.  These  types  of  impacts  are  particularly 
relevant  to  projects  that  are  environmental  in  nature  or  have  some  kind  of  a 
health  science  focus  since  action  is  a  desired  outcome. 

Skills 

Measurable  demonstration  of  the  development  and/or  reinforcement  of  skills, 
either  entirely  new'  ernes  or  the  reinforcement,  even  practice,  of  developing 
skills.  These  tend  to  be  procedural  aspects  of  blowing,  as  opposed  to  the 
more  declarative  aspects  of  knowledge  impacts.  Although  they  can 
sometimes  manifest  as  engagement,  typically  observed  skills  include  a  level 
of  depth  and  skill  such  as  engaging  m  scientific  inquiry  skills  (observing,, 
classifying,  exploring,  questioning,  predicting,  or  experimenting),  as  wrell  as 
developing/practicing  very  specific  skills  related  to  the  use  of  scientific 
instruments  and  devices  (e.g.  using  microscopes  or  telescopes  successfully). 

Other 

Project  specific. 
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Table  11.  YES  Short-term  Impacts 


Target 

Audience 

Impact 

Category 

Participant  Objectives 

YES  Teens 

Behavior 

Increasing  numbers  of  teens  will  stay  in  the  program  for  four  years 

Increasing  numbers  of  teens  will  graduate  from  high  school 

Increasing  numbers  of  teens  will  apply  to  and  be  accepted  by  two- 
and  four-year  colleges 

Increasing  numbers  of  teens  will  pursue  STEM-related  degrees  or 
careers 

Attitude 

Teens  will  become  comfortable  with  STEM  and  see  value  in  STEM 

Teens  will  become  comfortable  working  with  scientists 

Awareness, 

Teens  will  deepen  their  understanding  of  STEM  concepts 

Knowledge  or 
Understanding 

Teens  will  gain  exposure  to  many  STEM  academic  and  career 
pathways 

Teens  will  make  connections  between  YES  investigations  and 

STEM  concepts  studied  in  school 

SLSC  Program 
Staff 

Behavior 

Develop  new  opportunities  for  partnership  between  SLSC  staff 
and  Navy  personnel 

Become  advocates  for  teens  by  addressing  barriers  and  biases  in 
schools  and  communities 

Create  a  strategic  plan  and  budget  for  national  expansion 

Create  training  materials  for  staffs  at  other  institutions 

Attitude 

Develop  comfort  with  STEM  content 

Develop  a  culture  of  reflective  practice 

Awareness, 
Knowledge  or 
Understanding 

Strengthen  the  STEM  focus  of  programs 

Skills 

Develop  reflective  practice  skills 

Develop  the  ability  to  engage  youth  in  STEM  inquiry-based 
activities  and  investigations 

Develop  the  ability  to  train  and  support  others  in  building  similar 
programs  in  their  home  community 

YES  Alumni 

Behavior 

Increase  number  of  teens  who  graduate  from  college 

Increase  number  of  teens  who  pursue  STEM-related  careers 

Science  Center 
Professionals 

Skills 

Develop  skills  and  strategies  necessary  to  develop  STEM 
programs  built  on  the  SLSC  model  in  their  own  community 
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Table  11  shows  the  impact  statements  developed  for  the  YES  Program  and  used  to 
structure  the  program  evaluation.  Equally  as  important,  impacts  are  useful  in  making 
program  decisions.  For  example,  while  the  high  school  graduation  rates  of  YES  Teens 
appear  to  have  been  high  since  the  program  was  established,  some  staff  members 
observed  teens  struggled  with  college  admissions  tests,  college  applications,  and  finding 
funding  for  college.  Now  the  YES  Program  includes  College  Prep  learning  labs  to 
address  these  issues,  and  college  admission  rates  are  tracked  by  staff  members  through 
a  telephone  interview  with  seniors  each  year. 
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EVALUATION  AND  RESEARCH 


EVALUATION 

As  part  of  the  Community  STEM  Outreach  Project,  Klein  Consulting  conducted  both 
formative  and  summative  evaluation  studies.  Formative  evaluation  provides 
information  aimed  at  program  improvement;  the  audience  of  these  studies  generally 
consists  of  program  staff  members.  Formative  evaluation  findings  support  decisions 
about  changes  to  improve  the  program.  Evaluators  conducted  formative  evaluation 
during  Years  l  and  2  of  the  project.  Summative  evaluation  provides  information  about 
the  overall  value  and  effectiveness  of  the  program,  and  supports  decisions  about 
whether  the  program  should  continue  and  to  what  extent  the  program  as  a  whole  or 
elements  of  the  program  can  be  adopted  or  adapted  to  other  settings. 


PROGRAM  DOCUMENTS 


Klein,  C.,  &  Tisdal,  C.  (2012).  Community  STEM  Outreach  Second  Annual  Evaluation  Report. 

Saint  Louis  Science  Center,  unpublished  report. 

Klein,  C.,  &  Tisdal,  C.  (2013).  Community  STEM  Outreach  Summative  Evaluation.  Saint  Louis 
Science  Center,  unpublished  Report. 
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RESEARCH  AGENDA 

Programs  that  use  research  need  to  participate  in  research  projects  in  order  to 
contribute  to  the  body  of  knowledge  available  to  support  best  practices.  Participating  in 
research  projects  brings  new  ideas  and  practices  into  an  OST  program  and  helps  share 
the  research  agenda  for  the  knowledge -building  fields  it  draws  upon.  As  part  of  the 
Community  STEM  Outreach  Project,  the  YES  Program  developed  a  Research  Agenda 
that  is  available  on  the  Circles  of  Support. 


Klein  C.  &  Tisdal  C.  (2013).  Community  STEM  Outreach  Proposed  Research  Agenda  Report. 
Available  at  http:/ /www.yescirclesofsupport.com/ #  !research-agenda/cihxw. 
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Community  STEM  Outreach 
Summative  Evaluation  Report 


Executive  Summary 

The  summative  evaluation  report  provides  an  overview  of  the  Community 
STEM  Outreach  Project  at  the  Saint  Louis  Science  Center  (SLSC),  funded  by  the 
United  States  Office  of  Naval  Research  (ONR)  from  October  2010  through 
September  2013.  The  summative  evaluation  was  conducted  by  Klein  Consulting 
under  the  leadership  of  Christine  Klein,  Principal,  and  with  support  from  Carey 
Tisdal,  Director,  Tisdal  Consulting. 

The  evaluation  was  designed  to  address  three  questions: 

1 .  How  does  participation  in  the  Community  STEM  Outreach  Program  impact 
its  participants? 

2.  Does/how  does  the  Community  STEM  Outreach  Program  meet  its  goals 
and  objectives? 

3.  How  does  the  addition  of  50  or  more  teens  per  year  affect  the  program’s 
ability  to  meet  its  goals? 

To  aid  in  the  evaluation  and  program  documentation  process,  evaluators 
reviewed  existing  YES  Program  evaluations  and  related  documents.  Four 
themes  emerged  that  helped  guide  the  evaluation:  the  power  of  relationships, 
teaching  children,  STEM  knowledge  and  expertise  of  staff,  and  sustainability. 

Project  Context 

Shortly  after  the  Community  STEM  Outreach  Project  received  funding, 
changes  began  at  SLSC.  The  president,  who  had  been  in  place  since  the 
beginning  of  the  YES  Program,  resigned,  followed  by  an  interim  president  who 
served  until  a  new  president  was  hired.  During  this  period  of  transition,  the 
institution  underwent  financial  restructuring  and  reorganization  of  staff,  including 
many  layoffs.  During  the  last  year  of  the  project,  the  founder  and  leader  of  the 
YES  Program  resigned  to  take  a  position  at  another  institution.  Some  of  the 
effects  of  these  changes  are  included  in  the  evaluation  findings,  though  many 
effects  will  never  be  truly  understood. 

Program  Documentation  and  Dissemination 

In  its  original  proposal  to  the  ONR,  the  SLSC  proposed  to  “create  a  strategic 
plan  for  national  expansion  of  the  SLSC  Community  STEM  Outreach  Program" 
and  to  “identify  resources,  including  national  partners,  for  national  outreach.”  The 
original  intent  of  the  ONR  in  funding  the  Community  STEM  Outreach  Project  was 
to  document  a  mature,  successful  youth  development  program,  the  Youth 
Exploring  Science  (YES)  Program,  and  then,  with  additional  funding,  disseminate 
the  model  nationally  to  other  science  centers  and  museums.  The  trajectory  of  this 
initial  plan  was  established  to  benefit  SLSC,  ONR,  and  the  field  of  science 
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learning  in  out-of-school  time  (OST).  Through  the  Community  STEM  Outreach 
Project,  potential  partners  would  come  together  to  create  a  collaborative  and  plan 
for  the  dissemination  of  the  model,  including  application  for  a  new  ONR  grant. 

Several  environmental  factors,  leadership  and  staff  changes,  and  the  national 
economic  and  political  landscape,  influenced  the  need  to  adjust  the  initial 
trajectory.  As  part  of  the  revised  plan,  in  December  2012  SLSC  contracted  with 
Klein  Consulting,  in  collaboration  with  Tisdal  Consulting  to  document  the  YES 
Program  model  in  a  format  that  could  be  disseminated  nationally  as  part  of  the 
evaluation,  and  to  recommend  a  strategic  plan  for  dissemination.  Thus,  the 
evaluation  took  on  an  additional  focus.  The  result  is  the  design  of  a  multimedia 
tool,  currently  titled  Circles  of  Support,  described  in  the  full  summative  evaluation 
report. 

Deliberate  Design 

Throughout  each  step  of  the  Community  STEM  Outreach  Program,  the  initial 
project  PI,  Diane  Miller,  then  Vice  President  at  SLSC,  engaged  in  Deliberate 
Design,  an  approach  that  had  been  practiced  since  the  inception  of  the  YES 
program  in  1998.  With  Deliberate  Design  each  element  of  the  YES  Program  sits 
on  a  foundation  of  research  and  best  practices  in  youth  development,  STEM 
education,  and  Out-of-School  Time  (OST)  education.  Elements  of  the  program 
are  deliberate;  there  is  a  rationale  behind  each  feature  of  the  program.  The 
design  process  is  intentional,  such  that  each  program  element  aims  toward  an 
intended  impact. 

This  Deliberate  Design  served  as  the  foundation  of  the  evaluation  and  the 
multimedia  tool  developed  to  support  dissemination  of  the  YES  Program  model. 

Findings  -  Impact  on  Participants 

Evaluation  data  include  information  on  the  438  teens  participating  in  the  YES 
Program  over  the  past  three  years,  since  funding  from  the  Office  of  Naval 
Research  began.  We  defined  a  participating  teen  as  one  who  attended  at  least 
two  days  in  any  of  the  eight  semesters  since  the  beginning  of  the  ONR  project 
(Spring,  Summer  and  Fall  201 1  and  2012  plus  Spring  and  Summer  2013). 

Tables  and  figures  in  the  full  summative  report  are  based  on  these  438  teens. 

Demographic  data  included  information  on  gender  (53%  female),  ethnicity 
(87%  Black  or  African  American),  grade  level  in  school  (grades  7-12),  cohort  (a 
large  influx  in  201 1  with  ONR  funding),  and  school  type  (most  teens  from  public 
Missouri  schools).  Psychometric  data  included  why  teens  joined  the  YES 
Program  and  why  they  kept  coming  back  each  year.  The  top  three  reasons  for 
joining  were  related  to  the  work  experience  and  pay.  The  top  two  reasons  for 
returning  were  also  related  to  work  and  pay.  The  third  most  popular  reason  for 
returning  was  the  work  with  the  younger  children  -  teaching  them  science. 

Program  outputs  included  twenty-two  different  “components”,  or  groups 
focused  around  a  topic.  Components  focused  on  astronomy, 
biology/environmental  science,  chemistry,  engineering/design,  science 
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journalism,  and  teaching  science  to  younger  children,  which  was  an  additional 
component  oriented  toward  new  teens. 

Short-term  impacts  included  high  school  graduation  for  teens  who  completed 
their  senior  year  in  the  YES  Program  at  a  rate  higher  than  a  weighted  average  for 
teens  from  the  same  area  schools  and  districts.  Figure  10  in  the  summative 
report  depicts  the  difference  between  the  comparison  group  (Area  Students)  and 
the  YES  seniors. 
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Figure  10.  2012  and  2013  High  School  Graduation  Rate  Comparisons 

As  described  in  the  full  summative  report,  comparison  of  high  school 
graduation  rates  shows  that  YES  Teens  who  remain  in  the  program  until  their 
senior  year  in  high  school  graduate  at  a  higher  rate  than  their  peers  in  the  same 
area  schools.  Critical  is  the  phrase  “YES  Teens  who  remain  in  the  program  until 
their  senior  year  in  high  school.”  School  district  and  state  graduation  rates  must 
include  students  who  began  as  freshmen  but  dropped  out  of  school  before  or 
during  their  senior  year  (and  did  not  transfer).  The  YES  Program  does  not  keep 
attrition  data  and  does  not  follow-up  with  teens  that  drop  out  of  the  program  to 
determine  whether  or  not  they  graduated  from  high  school.  Thus,  when 
comparing  high  school  graduation  rates  between  YES  seniors  and  weighted 
averages  for  area  schools  or  states,  the  differing  definitions  of  graduation  should 
be  noted. 

Data  on  post-secondary  plans  of  graduating  seniors  showed  that  55%  in  2012 
and  46%  in  2013  planned  to  attend  a  four-year  college  or  university.  Additional 
teens  planned  to  attend  two-year  colleges,  trade  schools,  and  art  institutes, 
resulting  in  82%  in  2012  and  70%  in  2013  continuing  their  post-secondary 
education.  Two  teens  in  2012  and  one  in  2013  joined  the  military.  Details  are  in 
included  in  the  full  summative  report. 
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With  teens  in  the  YES  Program  for  four  and  a  half  or  more  years,  long-term 
impacts  include  those  impacts  recognized  after  they  have  left  the  program  and 
gone  on  to  college,  the  military,  and/or  careers.  To  begin  to  measure  the  long¬ 
term  impact  of  the  program,  an  online  survey  was  developed  for  former  YES 
participants  asking  about  their  experiences  during  and  after  YES.  Twenty-six 
individuals  participated  in  the  survey  by  the  deadline,  with  22  surveys  completed 
by  former  YES  Teens  to  the  extent  that  they  could  be  included  in  the  results 
reported  (N=22).  The  small  number  of  responses  can  give  a  limited  picture  of  the 
activities  of  YES  alumni/alumnae,  but  caution  should  be  exercised  in  drawing  any 
conclusions  about  program  impact. 

When  asked  how  well  the  YES  Program  prepared  them  for  post-secondary 
education,  alumni  indicated  that  the  program  was  “helpful”  (3.80  average  on  a  5- 
point  scale  with  5  =  greatly  helpful).  When  asked  how  well  the  program  prepared 
them  for  the  workforce,  the  resulting  average  was  “very  helpful”  (4.53  out  of  5). 
When  asked  how  influential  the  YES  Program  was  on  college  and  career 
choices,  the  average  was  3.88. 

Results  indicate  that  networking, 
teambuilding,  and  teaching  younger 
children  were  the  aspects  of  the 
program  that  influenced  respondents 
the  most.  When  asked  a  slightly 
different  question,  “To  what  extent  did 
these  elements  of  the  YES  Program 
positively  impact  you?”  rated  from  1 
(no  positive  impact)  to  5  (high 
positive  impact),  teaching  younger 
children  and  developing  job  skills 
received  the  highest  averages, 
followed  by  earning  an  income  and 
public  speaking  experiences. 

To  measure  changes  in  current 
YES  Teens’  attitudes  toward  science  and  scientists,  evaluators  used  a  modified 
version  of  the  Test  of  Science-Related  Attitudes  (TOSRA)  (Fraser,  1981) 
developed  by  Ledbetter  &  Nix  (2002)  -  the  TOSRA2. 

The  seven  subscales  were  as  follows: 

•  Social  Implications  of  Science  (S)  -  Do  youth  recognize  the  benefits  and 
drawbacks  of  scientific  advances  to  society? 

•  Normality  of  Scientists  (N)  -  Do  youth  see  scientists  as  real  people  rather 
than  media-produced  stereotypes? 

•  Attitude  toward  Scientific  Inquiry  (I)  -  Do  youth  view  experimentation  and 
inquiry  as  a  way  to  gain  understanding  of  the  natural  world? 

•  Adoption  of  Scientific  Attitudes  (A)  -  Have  youth  adopted  the  attitudes  of 
scientists,  such  as  open-mindedness  and  self-assessment? 


I  am  forever  grateful  to  the  YES 
program.  I  love  the  program. 
The  YES  program  influenced 
the  work  that  I  put  forth  in  other 
positions  I've  held  since  leaving 
the  program.  It  taught  me  work 
ethic  and  organizational  skills 
that  I'll  never  forget. . ..  My 
participation  in  the  YES 
program  helped  me  stay  in  the 
sciences.  It  kept  me  motivated 
and  kept  me  loving  science  and 
mathematics.  (1999-2003, 
female) 
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•  Enjoyment  of  Science  Lessons  (E)  -  To  what  degree  do  youth  enjoy  their 
lessons  in  school  science  classes? 

•  Leisure  Interest  in  Science  (L)  -  To  what  degree  are  youth  interested  in 
science  out  of  school,  and  outside  of  the  YES  Program? 

•  Career  Interest  in  Science  (C)  -  Do  youth  have  an  interest  in  pursuing  a 
science  related  career? 

The  matched  pairs  used  for  the  analysis  were  scores  from  teens  responding 
to  both  the  2012  spring  pre-test  and  the  2013  summer  post-test  (N=44),  which 
provided  the  greatest  time  between  tests.  As  described  in  more  detail  in  the  full 
summative  report,  results  suggest  that  respondents’  attitudes  changed  in  the 
following  ways.  After  16  additional  months  in  the  YES  Program  (i.e. ,  16  months 
between  testing),  youth  were: 

•  More  likely  to  recognize  the  benefits  and  drawbacks  of  scientific  advances 
to  society  --  Social  Implications  of  Science  (S) 

•  More  likely  to  see  scientists  as  real  people  rather  than  media-produced 
stereotypes  -  Normality  of  Scientists  (N) 

•  Less  likely  to  view  experimentation  and  inquiry  as  a  way  to  gain 
understanding  of  the  natural  world  (gender  differences  are  described  in 
the  report)  -  Attitude  toward  Scientific  Inquiry  (I) 

•  Slightly  more  likely  to  have  adopted  the  attitudes  of  scientists,  though  the 
adoption  of  attitudes  was  relatively  weak  (i.e.  not  significantly  different)  - 
Adoption  of  Scientific  Attitudes  (A) 

•  More  likely  to  indicate  enjoyment  of  their  lessons  in  school  science 
classes  -  Enjoyment  of  Science  Lessons  (E) 

•  More  interested  in  science  out  of  school,  and  outside  of  the  YES  Program 
-  Leisure  Interest  in  Science  (L) 

•  Slightly  less  interested  in  pursuing  a  science  related  career  (gender 
differences  are  described  in  the  report)  -  Career  Interest  in  Science  (C) 

Overall  test  scores  showed  a  significant  increase  from  pre-test  to  post-test. 

The  evaluators  explored  career  choices  of  the  YES  Teens  through  multiple 
methods.  From  the  surveys,  TOSRA2,  focus  groups,  and  interviews,  we  found 
that  while  anecdotal  evidence  existed  and  stories  were  available,  the  vast 
majority  of  YES  Teens  either  entered  the  program  with  career  interests  in  mind 
and  retained  those  interests,  or  entered  the  program  with  no  clear  career  path  in 
mind  and  left  with  remaining  uncertainty.  The  YES  Program  did  expose  teens  to 
careers  that  they  had  not  previously  considered,  and  it  was  not  clear  how  many 
teens  were  influenced  by  that  exposure.  Further  studies  of  former  program 
participants  would  be  needed  to  determine  that  impact. 

Discussion  -  Goals  and  Objectives  Met 

The  Community  STEM  Outreach  Program  met  many  of  the  project’s  goals 
and  objectives.  Personnel  and  budget  changes  at  SLSC  created  challenges  for 
meeting  others,  as  described  in  more  detail  in  the  full  summative  evaluation 
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report.  Of  the  nine  goals  and  14  objectives,  results  indicate  that  the  YES 
Program: 

•  Expanded  the  YES  Program  to  reach  more  youth,  though  the  increase 
was  not  sustained 

•  Increased  diversity  of  the  youth 

•  Strengthened  STEM  content  focus 

•  Involved  Navy  personnel  in  the  program  as  volunteers,  though  their 
participation  was  not  systematized  for  ongoing  engagement  or  to  the 
extent  that  it  could  be  replicated 

•  Involved  members  of  the  science  community  in  STEM  education 
programming,  though  this  engagement  was  not  systematized  into  an 
ongoing  effort  or  to  the  extent  that  it  could  be  replicated 

•  Strengthened  the  reflective  practice  of  YES  staff  and  SLSC  educators 
through  additional  training  and  ongoing  support. 

•  Supported  the  development  of  the  Circles  of  Support  multimedia  tool 

•  Supported  the  evaluation  to  provide  evidence  of  the  success  of  the 
program  and  to  identify  challenges 

•  Held  a  meeting  of  representatives  from  nine  science  centers  and 
museums  across  the  country. 

The  full  summative  report  provides  further  information  on  goals  and  objectives 
met  and  those  not  met. 

Discussion  -  Impact  of  50  Additional  Teens 

The  addition  of  teens  appeared  to  have  required  more  formalized 
management  structures  and  practices,  such  as  increased  manager  time  in 
scheduling  and  logistics.  The  larger  number  of  teens  increased  the  need  for  (1) 
formal  professional  development  for  consistent  implementation  of  the  Deliberate 
Design,  (2)  additional  curriculum  development,  (3)  consistent  data  management, 
and  (4)  renewed  support  for  youth  participation. 

Conclusion 

The  Deliberate  Design  of  the  YES  Program  lays  a  strong  foundation  for  youth 
development.  Many  positive  impacts  were  found:  higher  than  average  high 
school  graduate  rates,  large  numbers  of  teens  planning  to  continue  their 
education  beyond  high  school  graduation,  and  alumni/alumnae  who  stated  that 
the  YES  Program  was  helpful  in  preparing  them  for  post-secondary  education 
and  very  helpful  in  preparing  them  for  the  workforce.  Participants  improved  their 
attitudes  toward  science  and  scientists. 

At  the  same  time,  the  program  was  challenged  by  personnel  turnover, 
sometimes  resulting  in  inconsistent  application  of  the  Deliberate  Design  of  the 
program.  Thus,  we  learned  through  the  evaluation  that  consistent  systems  of 
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professional  development  and  sufficient  managerial  staffing  are  needed  to 
maintain  the  Deliberate  Design.  In  addition,  we  learned  the  vulnerability  of 
program  functions  with  the  loss  of  institutional  memory  due  to  staff  turnover. 

The  YES  Program,  though  challenged  at  times  by  a  number  of  factors 
described  in  the  summative  evaluation,  offers  a  model  for  other  youth  STEM 
programs.  As  with  all  programs,  ongoing,  sustainable  funding  for  the  program 
requires  telling  the  YES  story  to  stakeholders.  Through  the  evaluation,  we  found 
that  the  development  and  maintenance  of  program  records,  which  is  key  to  this 
sustainability,  needs  to  be  a  focus  as  the  program  moves  forward. 

This  summative  report  provides  part  of  the  YES  story  at  a  moment  of  change 
and  challenge.  As  the  program  continues  to  grow  and  change,  this  evaluation 
team  recommends  that  the  Deliberate  Design  of  the  program  remains  a  solid 
foundation.  This  means  that  clear  rationale,  based  on  research  and  best 
practices,  needs  to  be  developed  for  any  changes  with  an  eye  toward  how  these 
changes  may  affect  impacts  documented  in  this  evaluation. 


Feedback  on  this  report  and  questions  about  the  evaluation  can  be  sent  to 
Christine  (Kit)  Klein,  evaluation  consultant,  at  ckleinconsutling@gmail.com. 
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Introduction  And  Background 

The  Community  STEM  Outreach  Project  at  the  Saint  Louis  Science  Center  (SLSC) 
received  funding  from  the  United  States  Office  of  Naval  Research  (ONR)  from  October 
2010  through  September  2013.  Klein  Consulting,  with  support  from  Tisdal  Consulting, 
conducted  the  evaluation  of  the  three-year  project.  This  summative  evaluation  report 
provides  an  overview  of  that  work. 

Overview  of  Community  STEM  Outreach  Project 

The  original  proposal  from  the  SLSC  to  ONR 
laid  the  foundation  for  the  Community  STEM 
Outreach  Project  by  describing  the  institution  (see 
sidebar)  and  its  youth  program,  the  Youth  Exploring 
Science  (YES)  Program.  Plans  were  underway  to 
reach  out  to  existing  and  new  national  partners  to 
document  and  disseminate  a  successful,  replicable, 
and  scalable  science,  technology,  engineering,  and 
math  (STEM)  focused  youth  program,  YES.  The 
Community  STEM  Outreach  Project  was  designed 
to  use  that  dissemination  “to  meet  the  Navy’s  goal 
of  outreach  to  the  best  prepared  and  brightest 
youth  who  will  serve  as  the  next  generation  of 
Naval  recruits  to  serve  our  country”  (SLSC,  2010). 

The  intent  was  not  to  use  the  program  to  recruit 
youth  to  join  the  Navy.  Instead,  the  idea  was  to  use 
the  YES  Program  model  to  create  a  national 
collaboration  and  comprehensive  approach  to 
develop  programs  nationally  that  were  “designed  to 
effectively  meet  the  nation’s  needs  for  the  next 
generation  of  STEM  experts  and  leaders”  (SLSC, 

2010),  whether  those  experts  and  leaders  are  in 
the  military,  in  STEM  fields  that  support  the  military, 
or  in  fields  that  support  the  nation  in  other  roles. 

Deliberate  Design 

Throughout  each  step  of  the  Community  STEM  Outreach  Program,  the  initial  project 
PI,  Diane  Miller,  then  Vice  President  at  SLSC,  engaged  in  Deliberate  Design,  an 
approach  that  had  been  practiced  since  the  inception  of  the  YES  program  in  1998.  With 
Deliberate  Design  each  element  of  the  YES  Program  sits  on  a  foundation  of  research 
and  best  practices  in  youth  development,  STEM  education,  and  Out-of-School  Time 
(OST)  education.  Elements  of  the  program  are  deliberate;  there  is  a  rationale  behind 
each  feature  of  the  program.  The  design  process  is  intentional,  such  that  each  program 
element  aims  toward  an  intended  impact. 

This  Deliberate  Design  served  as  the  foundation  of  the  evaluation  and  the 
multimedia  tool  developed  to  support  dissemination  of  the  YES  Program  model. 
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Saint  Louis  Science  Center 

The  mission  of  the  Saint  Louis 
Science  Center  (SLSC)  is  “to  ignite 
and  sustain  lifelong  science  and 
technology  learning.”  Recognizing 
that  science  literacy  is  fundamental 
to  national  success  and  global 
competition  in  the  21st  Century,  the 
SLSC  promotes  high-quality 
education  in  science,  technology, 
engineering  and  mathematics 
(STEM).  Offering  general  admission 
at  no  charge  to  the  public,  SLSC 
serves  more  than  one  million  visitors 
annually.  In  addition  to  the  hands-on 
learning  opportunities  provided 
through  interactive  exhibits  and 
innovative  galleries,  SLSC  extends 
award-winning  educational  outreach 
programming  to  more  than  200,000 
students,  teachers  and  other 
individuals  within  the  bi-state  region 
annually. 
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Project  Goals  and  Objectives 

In  the  original  proposal  to  the  ONR,  SLSC  included  three  “central  components”  to  be 
carried  out  over  the  three-year  project,  with  nine  project  goals,  and  14  project 
objectives.  The  evaluation  focused  largely  on  the  first  component  since  program  staff 
focused  their  work  on  the  YES  Program  expansion  rather  than  outreach  to  schools  and 
scientists  or  national  expansion. 


Central  Project  Components 

1 .  Expansion  of  the  Youth  Exploring  Science  (YES)  Program  at  SLSC  by  an 
additional  50  teens  and  related  expansion  of  SLSC  staff 

2.  Expansion  of  Outreach  to  the  Community  to  additional  high  school 
teachers  and  counselors,  scientists,  Navy  personnel,  and  science  related 
members  of  the  community 

3.  Exploration  and  implementation  of  National  Expansion  initiatives  of  the 
YES  Program 


Community  STEM  Outreach  Project  goals  2-9  received  less  attention  by  project  staff 
than  goal  1 ,  though  all  were  reviewed  by  project  leaders  quarterly.  Each  goal  was 
included  in  the  evaluation,  and  is  addressed  in  the  Discussion  section  of  this  report. 


Project  Goals 

1 .  Expand  the  YES  Program  to  reach  more  youth,  increase  diversity  of  the 
youth,  and  strengthen  STEM  content  focus 

2.  Expand  the  SLSC  outreach  to  schools,  particularly  high  school  science 
teachers  and  school  counselors,  to  reach  more  students  and  teachers  and 
to  develop  new  models  of  outreach 

3.  Create  a  new  model  for  partnership  with  the  US  Navy  to  include  veterans 
as  well  as  active  duty  and  reserve  personnel  in  STEM  education  outreach 
programming 

4.  Create  and  formalize  a  model  for  outreach  to  and  inclusion  of  members  of 
the  science  community  in  STEM  education  programming 

5.  Improve  reflective  practice  of  all  educators  at  SLSC  to  create  a  cadre  of 
leaders  for  national  outreach  to  other  science  centers 

6.  Formalize  processes  and  collect  metrics  to  measure  YES  Program  short¬ 
term  and  long-term  success 

7.  Codify  a  system  for  community  STEM  outreach  beyond  YES 

8.  Conduct  research  and  evaluation  to  provide  evidence  of  the  success  of  the 
program  and  to  identify  challenges 

9.  Create  a  strategic  plan,  resources  and  model  for  national  expansion  of  the 
SLSC  Community  STEM  Outreach  Program  and  begin  implementation 
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Project  objectives  further  clarified  the  intent  of  project  leaders.  Midway  through  the 
project,  objectives  10  - 13  were  assigned  to  the  evaluation  team  at  Klein  Consulting. 
Using  the  results  of  the  evaluation,  the  evaluation  team  created  a  design  for  the 
multimedia  documentation  of  the  YES  Program  model,  conducted  the  evaluation, 
identified  research  questions,  and  developed  a  strategic  plan  for  expansion  using  the 
multimedia  documentation.  All  objectives  are  addressed  in  the  Discussion  section  of  this 
report. 


Project  Objectives 

1 .  Increase  the  number  of  teens  participating  in  the  SLSC  YES  Program  by 
50 

2.  Increase  the  staff  size  beginning  to  support  the  increase  in  YES  teens, 
community  outreach,  and  other  expansion  efforts 

3.  Increase  the  number  of  community  partnering  organizations  to  include 
organizations  in  St.  Louis  County  with  outreach  to  more  diverse  youth 

4.  Reach  new  school  audiences  with  existing  and  new  SLSC  programming, 
focusing  on  school  counselors  and  high  school  science  teachers 

5.  Strengthen  the  STEM  content  focus  of  YES  components  to  include 
stronger  emphasis  on  STEM  in  existing  components  and  addition  of  new 
components  with  content  relevant  to  the  US  Navy 

6.  Develop  new  opportunities  for  partnership  between  SLSC  staff  and  Navy 
personnel  to  support  the  YES  Program  and  other  SLSC  outreach  activities 

7.  Develop  and  formalize  opportunities  for  involvement  of  practicing  and 
retired  scientists  in  the  community 

8.  Strengthen  the  reflective  practice  of  YES  staff  and  SLSC  educators 
through  additional  training  and  ongoing  support 

9.  Train  and  support  a  cadre  of  STEM  education  leaders  who  can  train  others 
in  effective  strategies  to  build  programs  that  are  community  relevant,  youth 
development  focused,  and  strong  in  STEM  content 

10.  Create  electronic,  multimedia  documentation  of  all  Community  STEM 
Outreach  activities  and  staff  reflections  for  support  of  expansion  efforts 

1 1 .  Conduct  evaluation  to  support  and  provide  evidence  of  the  success  of  the 
program  and  to  identify  challenges 

12.  Identify  research  questions  related  to  the  Community  STEM  Outreach,  and 
create  strategies  for  moving  forward  with  that  research,  including  seeking 
additional  funding  for  such  research 

13.  Create  a  strategic  plan  for  national  expansion  of  the  SLSC  Community 
STEM  Outreach  Program 

14.  Identify  resources,  including  national  partners,  for  national  outreach 


Klein  Consulting 


3 


A96 


During  the  course  of  the  project,  Objective  6,  “Develop  new  opportunities  for 
partnership  between  SLSC  staff  and  Navy  personnel  to  support  the  YES  Program  and 
other  SLSC  outreach  activities,”  was  dropped  from  some  reports  by  SLSC  to  ONR. 

Project  Context 

Shortly  after  the  Community  STEM  Outreach  Project  received  funding,  changes 
began  at  SLSC.  The  president,  who  had  been  in  place  since  the  beginning  of  the  YES 
Program,  resigned.  The  SLSC  Board  appointed  one  of  its  members  as  interim 
president,  and  during  the  second  year  of  the  project  a  new  president  was  hired.  During 
this  period  of  transition,  the  institution  underwent  financial  restructuring  and 
reorganization  of  staff,  including  staff  layoffs.  During  the  last  year  of  the  project,  the 
founder  and  leader  of  the  YES  Program  resigned  to  take  a  position  at  another 
institution.  Some  of  the  effects  of  these  changes  are  included  in  the  evaluation  findings, 
though  many  effects  are  likely  to  be  long-term  and  the  effects  were  not  apparent  during 
the  course  of  this  study. 


YES  Program  Overview 


The  Youth  Exploring  Science  (YES)  Program  began  in  1998  under  the  leadership  of 
Diane  Miller  at  the  SLSC  with  15  teenagers  meeting  in  a  small  warehouse  space  near 
SLSC.  By  the  beginning  of  the  Community  STEM  Outreach  Project,  the  program  looked 
very  different  with  216  teens.  Yet,  many  of  the  same  foundations  remained 


Then  as  now,  the  YES  Program  serves  St.  Louis 
area  teenagers  ages  14-18  in  a  work-based,  inquiry- 
based  learning  environment.  YES  Teens  are  recruited 
from  over  65  partnering  community  organizations 
committed  to  serving  low-income  families  in  the 
metropolitan  St.  Louis  area.  Through  the  four-plus 
year  program,  educators  strive  to  support  these  youth 
in  gaining  professional,  academic,  and  real  world  skills 
to  assist  them  in  building  self-confidence  and  personal 
success.  They  gain  exposure  to  STEM-related 
academic  and  career  pathways,  and  gain  experience 
through  their  work  for  SLSC. 


The  YES  Program  looks  very  different  in  the  school  year  and  summer.  Vignettes  of 
each  from  the  second  annual  evaluation  report  (Klein  and  Tisdal,  2012)  are  included  in 
Appendix  A  for  those  readers  unfamiliar  with  the  program.  During  the  school  year,  the 
focus  is  on  learning  the  STEM  content,  developing  workplace  and  21st  century  skills, 
college  prep  activities,  and  team  building.  Teens  then  apply  STEM  content  and  skills  in 
the  summer.  Thus,  the  primary  difference  between  the  school  year  and  summer  is  the 
shift  from  learning  to  teaching.  Teens  teach  others  in  the  summer,  with  community 
members  and  SLSC  visitors  as  their  audiences.  The  Deliberate  Design  of  the  YES 
Program  is  based  on  the  assumption  that  through  this  teaching  YES  Teens  deepen  their 
understanding  of  the  STEM  content  and  improve  their  21st  century  skills. 
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Figure  1  provides  an  overview  of  the  numbers  of  YES  Teens  participating  during 
each  semester  of  the  project,  with  numbers  from  2010  (prior  to  ONR  funding)  for 
comparison. 


Figure  1.  Number  of  YES  Teen  participants  by  semester 

The  lower  number  of  participants  in  each  fall  represents  the  loss  of  the  graduating 
seniors  at  the  end  of  each  summer,  while  the  increase  each  spring  represents  the 
addition  of  new  teens.  The  largest  increase  in  new  teens  was  in  spring  201 1  with  the 
beginning  of  the  ONR  funding.  That  year  1 06  new  teens  joined  the  program  instead  of 
the  typical  40-50.  Each  summer,  participation  increased  as  teens  returned  who  had  left 
the  program  during  the  academic  year  due  to  participation  in  sports  or  family  issues. 

Project  Stakeholders 

Stakeholders  for  the  Community  STEM  Outreach  Program  included  national 
partners,  local  community  groups,  and  individuals: 

•  YES  Teens  and  Their  Families 

•  Saint  Louis  Science  Center 

•  U.S.  Navy  and  Office  of  Naval  Research 

•  Science  Centers  and  Museums  Across  the  U.S. 

•  Educators  -  In  Schools  and  Out 

•  Community  Organizations 

•  Scientists 

Role  of  Evaluation  in  the  Project 

The  original  plan  for  the  evaluation  called  for  internal  and  external  evaluators  to  work 
together,  with  the  internal  team  focused  on  guiding  the  project  toward  success  by 
determining  how  to  improve  the  program  and  the  external  team  focused  on  providing 
evidence  of  overall  success  in  meeting  goals  and  objectives  to  prove  impacts.  As  the 
project  was  beginning,  this  plan  changed  when  the  internal  evaluator  moved  to  a 
different  role  within  the  SLSC.  Without  a  change  in  budget  for  the  evaluation,  the 
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external  evaluator  took  on  both  goals:  guiding  the  project  toward  success  (formative 
evaluation)  and  providing  evidence  of  success  (summative  evaluation).  (Evaluation 
activities  are  listed  in  Appendix  B,  data  sources  in  Appendix  C.) 

At  the  beginning  of  the  first  year  of  the  project,  the  evaluation  team  took  on  another 
role,  that  of  documenting  the  program.  Since  an  original  intent  of  the  ONR  in  funding  the 
Community  STEM  Outreach  Project  at  SLSC  was  to  document  a  mature,  successful 
youth  development  program,  i.e.  the  YES  Program,  it  was  decided  that  the  evaluation 
team  at  Klein  Consulting  was  well  suited  to  provide  that  documentation.  Beyond  the 
written  documentation,  the  team  provided  the  design  of  a  web-based  multimedia  tool  for 
disseminating  the  YES  Program  model,  as  described  in  later  sections. 

Results  of  the  formative  evaluation  efforts  can  be  found  in  previous  reports,  listed  in 
Appendix  B.  The  multimedia  tool  will  be  made  available  later  by  the  SLSC.  This  report 
provides  the  results  of  the  summative  evaluation. 

Research  regarding  impacts  of  the  YES  Program  was  originally  of  interest  to  the 
ONR,  though  funding  for  such  research  was  not  included  in  this  project.  An  additional 
role  of  the  evaluation  became  to  identify  research  questions  and  suggest  further 
studies.  The  potential  research  that  emerged  is  described  in  a  separate  document 
available  from  SLSC  or  the  authors. 

Thus,  the  purpose  of  the  evaluation  of  the  Community  STEM  Outreach  Project  was 
to  study  the  impact  of  the  program  on  participants,  determine  whether  and  how  the 
program  achieves  its  goals,  and  create  the  foundation  for  further  research. 

Definitions 

As  reported  previously,  a  few  definitions  are 
necessary  when  telling  the  YES  story  and  describing 
the  program  to  people  outside  of  the  program.  First, 
a  “component”  is  a  group  of  about  20  YES  Teens 
working  with  one  or  more  staff  members  on  a  STEM 
(science,  technology,  engineering  and  math)  topic. 

“New  Teens”  is  the  term  used  to  describe  the  YES 
Teens  during  their  first  spring  in  the  program  as  they 
learn  the  ropes.  The  group  “New  Teens”  is  generally 
referred  to  as  a  “component”  even  though  it  focuses  on  science  in  general  rather  than  a 
specific  STEM  content  area.  Teens  split  program  time  between  “components”  and 
“College  Prep.”  College  Prep  is  for  same-grade  groups  of  teens  to  work  with  staff  on 
aspects  of  college  planning  and  preparation.  This  group  of  semester-long  components 
and  college  prep  sessions  are  collectively  referred  to  as  Learning  Labs. 


Summative  Evaluation  Overview 

The  summative  evaluation  was  conducted  by  Klein  Consulting  under  the  leadership 
of  Christine  Klein,  Principal,  and  with  support  from  Carey  Tisdal,  Director,  Tisdal 
Consulting.  To  begin  to  understand  program  impacts,  naturalistic  methodology  was 
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used.  Through  that  approach  and  given  the  changing  context  of  the  project,  the  focus  of 
the  evaluation  became  the  YES  Program  rather  than  outreach  to  schools  and  scientists, 
though  all  goals  and  objectives  were  reviewed. 

Evaluation  Questions 

The  evaluation  was  designed  to  address  three  questions: 

1 .  How  does  participation  in  the  Community  STEM  Outreach  Program  impact  its 
participants? 

2.  Does/how  does  the  Community  STEM  Outreach  Program  meet  its  goals  and 
objectives? 

3.  How  does  the  addition  of  50  or  more  teens  per  year  affect  the  program’s  ability  to 
meet  its  goals? 

YES  Program  Logic  Model 

In  taking  a  close  look  at  the  YES  Program,  the  evaluation  team  used  the  program 
theory  logic  model  adapted  from  Weiss  (1998)  and  the  W.K.  Kellogg  Foundation  (2004) 
in  Figure  2  to  guide  the  evaluation. 
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Figure  2.  Program  Theory  Model 

This  logic  model  provides  the  framework  for  the  findings  of  the  summative 
evaluation. 

Impact  Matrix 

The  evaluation  was  guided  by  the  NSF  Frameworks  set  forth  by  the  NSF  Division  of 
Research  on  Learning  in  Formal  and  Informal  Settings  (DRL)  (Friedman  2008).  Table  1 
provides  the  impact  matrix  used. 

Table  1 .  Impact  Matrix  for  the  Community  STEM  Outreach  Project. 
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Target 

Audience 

Impact 

Category 

Participant 

Objectives 

Measures 

Methods 

YES  Teens 

Behavior 

Increasing  numbers  of 
teens  will  stay  in  the 
program  for  four  years 

Attendance 

records 

Document  analysis 

Increasing  numbers  of 
teens  will  graduate  from 
high  school 

Graduation  rates 

Survey  of  graduates 
by  YES  Staff  -  data 
analysis 

Increasing  numbers  of 
teens  will  apply  to  and 
be  accepted  by  two  and 
four  year  colleges 

College 
application  and 
acceptance 
rates 

Survey  of  graduates 
by  YES  Staff  -  data 
analysis 

Increasing  numbers  of 
teens  will  pursue  STEM 
related  degrees  or 
careers 

Self-report 
degree  and 
career  choice 

Questionnaire  data 
comparisons  by  year 

Attitude 

Teens  will  become 
comfortable  with  STEM 
and  see  value  in  STEM 

Attitude  survey 
&  observed 
behavior 

TOSRA2; 

Observations 

Teens  will  become 
comfortable  working  with 
scientists 

Self-report  & 

observed 

behavior 

Observations; 

Interviews 

Awareness, 
Knowledge  or 
Understanding 

Teens  will  deepen  their 
understanding  of  STEM 
concepts 

Interviews  & 

observed 

behavior 

Observations; 

Interactive 

Interviews 

Teens  will  gain  exposure 
to  many  STEM 
academic  and  career 
pathways 

Attendance  & 

participation 

records 

Document  analysis 

Teens  will  make 
connections  between 

YES  investigations  and 
STEM  concepts  studied 
in  school 

Self-report 

Interviews  and 

Focus  Groups 

SLSC 

Program 

Staff 

Behavior 

Develop  new 
opportunities  for 
partnership  between 

SLSC  staff  and  Navy 
personnel 

Self-report; 
records  of 
program 
offerings 

Document  analysis 

Become  advocates  for 
teens  by  addressing 
barriers  and  biases  in 
schools  and 
communities 

Self-report 

Interviews 

Create  a  strategic  plan 
and  budget  for  national 
expansion 

Completed  plan 
and  budget 

Document  analysis 
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Target 

Audience 

Impact 

Category 

Participant 

Objectives 

Measures 

Methods 

Create  training  materials 
for  staffs  at  other 
institutions 

Completed 

materials 

Document  analysis 

Attitude 

Develop  comfort  with 
STEM  content 

Self-report  and 

observed 

behavior 

Observations; 
Document  analysis 
of  meeting  notes 

Develop  a  culture  of 
reflective  practice 

Observed 

behavior 

Observations; 
Document  analysis 
of  meeting  notes 

Awareness, 
Knowledge  or 
Understanding 

Strengthen  the  STEM 
focus  of  programs 

Records  of 
program 
offerings  and 
observed 
activities 

Document  analysis; 
Observations 

Skills 

Develop  reflective 
practice  skills 

Observed  skills 
and  written 
reflections 

Observations; 
Document  analysis 
of  reflections 

Develop  the  ability  to 
engage  youth  in  STEM 
inquiry-based  activities 
and  investigations 

Comparisons  of 
observation  data 

Observations 

Develop  the  ability  to 
train  and  support  others 
in  building  similar 
programs  in  their  home 
community 

Observed  skills 
and  survey 
results  from 
those  trained 

Observations; 

Surveys 

YES  Alumni 

Behavior 

Increase  number  of 
teens  who  graduate  from 
college 

Self-report 

Surveys 

Increase  number  of 
teens  who  pursue  STEM 
related  careers 

Self-report 

Surveys 

Science 

Center 

Professionals 

Skills 

Develop  skills  and 
strategies  necessary  to 
develop  STEM  programs 
built  on  the  SLSC  model 
in  their  own  community 

Self-report  from 
participants  after 
return  home; 
observed  skills 
during  training 

Observations  of 
training;  Surveys  of 
participants  at  1,  3 
and  6  months  post 
training 

Prior  Evaluation  Report  Synthesis 

To  aid  in  the  evaluation  and  program  documentation  process,  evaluators  reviewed 
existing  YES  Program  evaluations  and  related  documents.  Evaluators  found  that  reports 
were  not  stored  in  one  central  location  and  no  one  staff  member  collected  them.  YES 
staff,  the  SLSC  Research  and  Evaluation  Department,  and  the  Development 
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Department  were  able  to  locate  the  following  reports.  These  were  used  in  the  analysis; 
however,  other  reports  may  exist. 

•  Summative  Evaluation  of  Designing  Youth:  Teens  Engaging  Children  in 
Design  Engineering  for  the  SLSC  by  Lorrie  J.  Beaumont  (2006) 

•  Teenage  Designers  of  Learning  Places  for  Children:  Creating  After-School 
Environments  for  STEM  Education  by  Christine  Klein  (2006) 

•  Learning  Places  Evaluation  -  Observations  of  Children  at  Torre:  An  Interim 
Report  by  Christine  Klein  (2007) 

•  Teenage  Designers  of  Learning  Places  for  Children:  Creating  After-School 
Environments  for  STEM  Education  -  Summative  Evaluation  by  Christine  Klein 
(2010) 

•  Science  Firsthand  -  Partners  in  Discovery  Year  One  Evaluation  Report  by 
Sabra  Lee  (2005) 

•  Science  Firsthand  -  Partners  in  Discovery  Year  Two  Evaluation  Report  by 
Sabra  Lee  and  Judah  Leblang  (2006) 

•  Science  Firsthand  -  Partners  in  Discovery  Year  3  Evaluation  Report  by  Sabra 
Lee  and  Judah  Leblang  (2007) 

•  Science  Firsthand  -  Partners  in  Discovery  Year  4  Evaluation  Report  by  Sabra 
Lee  and  Judah  Leblang  (2008) 

•  Science  Firsthand  -  Partners  in  Discovery  Year  5  Evaluation  Report  by  Sabra 
Lee  and  Judah  Leblang  (2009) 

•  Science  Firsthand  -  Partners  in  Discovery  Year  6  Evaluation  Report  by  Sabra 
Lee,  Judah  Leblang  and  Tracy  Wallach  (2011) 

•  YES-2 -Tech:  Youth  Exploring  Science  to  Technology  Year  3  Evaluation 
Report  by  Joseph  L.  Polman  (2007) 

•  Design  IT!  Building  Design  Challenges  in  After  School  Program  Final 
Evaluation  Report  by  Patricia  B.  Campbell,  Lesley  Perlman,  and  Earl  Hadley 
(2002) 

•  The  Legacy  of  YouthALIVE!  Transformative  Youth  Programs  Continue  to 
Thrive  in  Science  Centers  by  Cary  Sneider  and  Meg  Burke  (201 1 ) 

Evaluation  reports  from  the  Community  STEM  Outreach  project  were  not  included  in 
the  analysis  since  the  goal  was  to  inform  the  project  through  prior  evaluations. 

Additional  publications  relating  to  the  YES  Program  were  available,  but  were  not 
included  in  the  synthesis. 

Of  the  studies  included,  three  covered  national  programs  of  which  the  YES  Program 
was  one  of  several  sites  (Design  IT!,  Science  Firsthand,  and  the  Legacy  of 
YouthALIVE!)  Three  of  the  studies  focused  on  one  component  of  the  YES  Program 
(Designing  Youth,  Learning  Places,  and  YES-2-Tech).  All  included  qualitative  methods, 
some  included  additional  quantitative  methods. 

From  these  evaluation  reports  emerged  several  themes  that  informed  the  evaluation 
of  the  Community  STEM  Outreach  Program. 

The  power  of  relationships  stood  out  as  evaluators  described  the  interactions  among 
teens  and  between  teens  and  staff.  One  study,  Design  IT!  included  the  relationship 
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between  the  science  center  and  community  partners.  In  the  Designing  Youth  report, 
Beaumont  identified  the  characteristics  of  supportive  mentor  relationships  between 
adults  and  teens  as:  “warmth,  closeness,  connectedness,  good  communication,  caring, 
support,  guidance,  secure  attachment,  and  responsiveness”  (2006,  p.11)  The  Science 
Firsthand  project  focused  on  mentor/mentee  relationships  and  pointed  to  changes  in 
mentors  as  an  issue.  For  the  Community  STEM  Outreach  evaluation,  we  explored  the 
various  relationships.  For  example,  we  explored  the  impact  of  the  unanticipated  layoffs 
of  staff  with  established  relationships  with  teens  and  community  partners. 

Teaching  children,  which  we  are  calling  the 
Learn  to  Teach  -  Teach  to  Learn  strategy,  was 
described  as  a  central  component  in  the 
Designing  Youth  and  Learning  Places 
evaluations,  and  was  described  in  the  YES-2- 
Tech  report  as  a  summer  focus  of  the 
program.  In  each  case,  the  value  of  teens 
teaching  younger  children  was  described,  as 
were  challenges.  The  primary  issue  raised 
was  the  understanding  of  the  STEM  concepts 
by  the  teens  at  a  level  suitable  to  teach  others. 

If  the  goal  was  to  teach  STEM  concepts  to 
younger  children,  then  how  could  the  teens 
successfully  teach  something  they  didn’t  fully  u 
On  the  other  hand,  if  the  goal  was  to  engage  children  in  STEM  activities  to  show  them 
that  STEM  could  be  fun,  the  teens  were  well  placed  to  do  so.  For  the  Community  STEM 
Outreach  evaluation,  we  explored  the  way  learning  to  teach  STEM  concepts  to  younger 
children  impacted  YES  Teens’  understanding  of  STEM  concepts. 

STEM  Knowledge  and  Experience  of  Staff  was  called  into  question  by  some 
evaluations.  As  the  evaluation  of  Designing  Youth  found,  the  STEM  understanding  of 
educators  varied  widely.  At  the  same  time,  they  found  that  staff  grew  confident  in  their 
ability  to  engage  youth  in  inquiry  investigation.  In  the  Learning  Places  evaluation,  one 
theory  that  emerged  from  the  Grounded  Theory  approach  stated: 

Guiding  children  and  teens  in  investigations  to  create  rich  STEM  experiences 
requires  after-school  program  educators  who  understand  inquiry  and  are 
comfortable  with  the  STEM  content  and  materials.  Guiding  those  educators  to  lead 
such  experiences  requires  additional  personnel,  in  this  case  at  the  museum,  who 
have  the  skills  to  train  educators  in  leading  investigations  and  who  have  a  high 
degree  of  STEM  comfort  themselves.  (Klein,  2010,  p  54) 


By  the  time  of  the  Community  STEM  Outreach  evaluation,  the  YES  Program  leaders 
had  begun  hiring  educators  with  strong  STEM  backgrounds  as  part  of  the  Deliberate 
Design.  Some  of  these  new  educators  came  with  Masters  level  science  degrees; 
another  was  an  engineer. 

The  sustainability  of  the  projects  was  described  in  the  summative  reports.  In  each 
case,  evaluators  described  how  the  project  would  be  incorporated  into  the  YES 
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Program.  The  inclusion  of  lessons  learned  and  best  practices  from  these  projects,  then, 
became  part  of  the  foundation  for  the  Deliberate  Design  of  future  iterations  of  YES.  For 
the  Community  STEM  Outreach  evaluation,  we  explored  the  foundations  of  the  program 
model. 

Methodology  and  Methods 

Methodology 

Naturalistic  methodology  (Lincoln  &  Guba,  1985;  Guba  &  Lincoln,  1989)  was  the 
overarching  methodology  used  to  design  the  study  and  develop  conclusions.  Evaluators 
collected  and  analyzed  data  using  a  variety  of  qualitative  and  quantitative  methods.  In 
order  to  develop  metrics  for  key  indicators  of  success,  evaluation  was  guided  by  the 
National  Science  Foundations’  (NSF)  Frameworks  set  forth  by  the  NSF  Division  of 
Research  on  Learning  in  Formal  and  Informal  Settings  (DRL)  (Friedman,  2008). 

Responsive  constructivist  evaluation  using  naturalistic  methodology  aims  to  provide 
a  holistic  understanding  of  phenomena  by  looking  at  it  from  several  angles  in  a  real-life 
setting  using  a  systematic  approach  for  collecting  and  analyzing  data  in  the  context  in 
which  it  occurs.  In  responsive  constructivist  evaluation,  processes  and  activities  are 
captured  through  a  variety  of  sources  from  multiple  perspectives  of  various  stakeholder 
groups,  and  presented  through  deep  descriptions.  The  impacts  of  the  program  are 
captured  through  this  process,  and  are  connected  to  these  processes  and  activities 
through  the  viewpoints  and  perspectives  of  the  people  involved.  In  responsive 
constructivist  evaluation,  data  collection  and  analysis  are  iterative  processes.  This 
provided  stronger  validity  and  credibility  for  decision-making  and  allowed  decision 
makers  to  understand  how  conclusions  were  reached  and  the  evidence  upon  which 
they  were  based. 

Methods 

Methods  to  identify  key  project  issues  and  outcomes  and  to  assess  project  impacts 
included:  surveys,  focus  group  interviews,  information  interviews  of  key  project 
stakeholders,  in-depth  interviews  of  selected  participants,  interactive  interviews  with 
selected  youth,  document  analyses,  observations,  and  the  Test  of  Science-Related 
Attitudes  (TOSRA). 

Surveys  were  completed  by  YES  Teens  each  semester.  Data  included:  schools 
attended;  high  school  STEM  courses  taken;  plans  for  high  school  graduation,  college 
and  career;  mentors  and  their  post-secondary  experiences;  and  feedback  on  the  teens’ 
YES  experiences.  A  pilot  survey  of  YES  graduates  explored  trends  in  college 
graduation  and  career  choices. 

Focus  Group  Interviews  with  YES  Teens  and  community  partners  allowed  the 
evaluation  team  to  ask  focused  questions  of  groups  of  8-12  participants  in  each 
category.  Focus  groups  of  YES  Teens  provided  data  on  teens’  attitudes,  interests,  and 
understanding.  Focus  groups  of  community  partners  who  recruited  teens  into  the 
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program  provided  additional  perspectives  on  the  value  of  the  program  to  the  teens  and 
community. 

Information  Interviews  with  representatives  of  key  stakeholder  groups  allowed  the 
evaluation  team  to  collect  perceptions  and  opinions  and  was  used  to  identify  issues  and 
patterns.  These  interviews  laid  the  foundation  for  additional  interviews. 

In-depth  Interviews  of  selected  participants  provided  data  on  participant  attitudes, 
beliefs,  and  behaviors.  Interviews  of  staff  provided  data  on  the  SLSC-Navy  partnership, 
activities  involving  scientists,  staff  comfort  with  STEM  content,  and  their  reflective 
practice. 

Interactive  Interviews  with  selected  youth  were  used  to  identify  the  depth  of  their 
understanding  of  the  inquiry  process.  These  interviews  built  on  techniques  developed  in 
the  Learning  Places  project  in  which  teens  were  asked  to  engage  in  new  inquiry 
activities  to  determine  transfer  of  understanding  of  the  STEM  concepts  covered  in  their 
program. 

Document  Analysis  of  attendance  records  of  teens  provided  evidence  of 
participation  in  the  program  and  additional  opportunities  to  identify  trends.  Document 
analysis  of  curriculum  templates,  recruiting  materials,  and  other  program 
provided  evidence  for  the  triangulation  of  findings. 

Observation  strategies  built  on  methods 
and  techniques  adapted  from  previous 
observational  research  (Polman,  2000,  2004). 

The  evaluation  team  collected  observational 
data  in  multiple  locations  that  constituted  the 
learning  community  of  the  project.  The 
observational  data  included  direct  observation, 
field  notes,  and  selected  videotaped  episodes 
of  educational  activities  involving  youth,  staff, 
scientists,  and  OST  educators  in  after-school  sites,  as  well  as  the  professional 
development  activities  and  YES  Program  facilitation  at  the  SLSC. 

Test  of  Science-Related  Attitudes  (TOSRA)  (Fraser  1981)  was  designed  to  measure 
secondary  science  students’  attitudes  toward  science.  The  original  test  consisted  of  70 
statements  with  seven  subscales  using  a  5-point  Likert  Scale  (strongly  agree,  agree,  not 
sure,  disagree,  and  strongly  disagree).  TOSRA  has  been  used  with  youth  around  the 
world,  and  has  been  shown  to  be  valid  and  reliable  for  American  teens.  A  modified 
version  (TOSRA2)  developed  by  Ledbetter  and  Nix  (2002)  was  used  in  this  study  to 
reduce  the  time  needed  for  completion.  TOSRA2  consisted  of  35  pre-test  items  and  35 
post-test  items  with  negatively  and  positively  phrased  items  balanced  on  each  test.  The 
same  seven  subscales  were  used:  Social  Implications  of  Science;  Normality  of 
Scientists;  Attitude  toward  Scientific  Inquiry;  Adoption  of  Scientific  Attitudes;  Enjoyment 
of  Science  Lessons;  Leisure  Interest  in  Science;  and  Career  Interest  in  Science. 

Data  Analysis 

For  this  study,  Klein  Consulting  analyzed  qualitative  data  using  a  modified  inductive 
constant  comparison  approach  (Lincoln  &  Guba,  1985),  whereby  each  set  of  data  was 
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compared  with  previous  data  sets  to  direct  the  focus  of  subsequent  data  collection. 
Quantitative  data  was  initially  analyzed  using  descriptive  and  inferential  statistics.  In 
order  to  develop  findings  and  draw  conclusions  both  qualitative  and  quantitative  data 
were  triangulated. 

TOSRA2  scores  were  recorded  in  an  Excel  spreadsheet  and  imported  into  an  SPSS 
program  for  analysis.  Scores  for  each  of  the  seven  categories  and  a  total  were 
calculated  for  each  respondent  on  each  test.  Analysis  was  conducted  using  descriptive 
statistics,  two-tailed  t-tests,  one-way  ANOVAs,  and  Pearson  Correlations  on  matched 
pairs.  In  this  analysis,  matched  pairs  were  scores  of  the  same  teens  taking  both  the 
2012  spring  pre-test  and  the  2013  summer  post-test.  This  provided  the  greatest  time 
between  tests. 

Program  Documentation  and  Dissemination 

In  its  proposal  to  the  Office  of  Naval  Research  (ONR)  dated  August  4,  2010,  the 
Saint  Louis  Science  Center  (SLSC)  proposed  to  “create  a  strategic  plan  for  national 
expansion  of  the  SLSC  Community  STEM  Outreach  Program"  and  to  “identify 
resources,  including  national  partners,  for  national  outreach.”  The  original  intent  of  the 
ONR  in  funding  the  Community  STEM  Outreach  project  at  SLSC  was  to  document  a 
mature,  successful  youth  development  program,  the  Youth  Exploring  Science  (YES) 
Program,  and  then,  with  additional  ONR  funding,  disseminate  the  model  nationally  to 
other  science  centers  and  museums.  The  trajectory  of  this  initial  plan  was  established  to 
benefit  SLSC,  ONR,  and  the  field  of  science  learning  in  out-of-school  time  (OST). 
Through  the  Community  STEM  Outreach  Project,  potential  partners  would  come 
together  to  create  a  collaborative  and  plan  for  the  dissemination  of  the  model,  including 
application  for  a  new  ONR  grant. 

Several  environmental  factors,  leadership  and  staff  changes,  and  the  national 
economic  and  political  landscape,  influenced  the  need  to  adjust  the  initial  trajectory. 
After  hiring  part-time  summer  staff  to  document  the  program  in  201 1 ,  results  fell  short  of 
expectations,  and  SLSC  contracted  with  Klein  Consulting,  in  collaboration  with  Tisdal 
Consulting,  to  document  the  YES  Program  model  in  a  format  that  could  be 
disseminated  nationally  as  part  of  the  evaluation,  and  to  recommend  a  strategic  plan  for 
dissemination.  The  result  is  the  design  of  a  multimedia  tool,  currently  titled  Circles  of 
Support  (www.yescirclesofsupport.com). 

Circles  of  Support 

The  Circles  of  Support  web-based  multimedia  tool  design  brings  together  audio, 
video,  photos,  and  text  to  share  the  story  of  the  YES  Program.  Organized  around  the 
YES  Program’s  logic  model  and  growing  out  of  the  evaluation  of  the  YES  Program,  the 
tool  design  allows  users  to  look  at  the  broad  ideas  that  serve  as  the  program’s 
foundation  and  to  zero  in  on  specific  program  aspects,  like  the  learn  to  teach  -  teach  to 
learn  philosophy. 
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This  multimedia  tool  was  designed  to  meet  two  needs:  to  strengthen  the  existing 
YES  Program  at  SLSC  by  creating  a  record  of  the  program’s  underlying  rationale  for 
new  staff,  and  to  provide  the  creation  of  a  tool  to  support  expanded  opportunities  for 
youth  nationally  based  on  the  YES  Program  model.  The  tool  presents  elements  of  the 
YES  Program  as  a  working  model  to  allow  other  institutions  to  select  and  adapt  the 
ideas,  research,  and  best  practices 
that  fit  the  context  of  their  local  area. 

Target  audiences  for  the 
tool  are  community-based 
organizations  and  science  centers 
interested  in  starting  a  new 
program  or  improving  an  existing 
program.  Units  about  each 
program  element  include  a  video 
introduction,  documentation  of  the 
YES  Program  element,  sample 
program  documents,  and 
references.  An  alpha  version  of  the 
tool,  available  through  a  website, 
was  available  for  testing  at  the  conclusion  of  the  grant  period  along  with  a  print 
documentation  available  in  PDF  on  the  website  or  by  request. 

Dissemination 

The  proposed  dissemination  plan  involved  three  phases:  1)  testing  and  revision  of 
the  Circles  of  Support  multimedia  tool  design;  2)  broad  dissemination  of  the  URL  for  the 
website;  and,  3)  professional  development  of  staff  using  the  website  and  related 
resources  as  a  foundation.  To  distribute  the  URL  for  the  website  after  any  necessary 
revisions,  it  was  recommended  that  SLSC  utilize  the  Science  Beyond  the  Boundariessm 
network,  ISEN  listserv,  ASTC’s  Youth  Program  Network,  and  the  ONR  Stem2Stern 
program.  At  the  time  of  the  writing  of  this  summative  evaluation,  it  was  too  early  to 
determine  what  path  SLSC  would  take  for  disseminating  the  results. 


Circles  of  Support 
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Use  this  website  to  translate  current  research,  theory, 
and  best  practices  for  community-based  STEM  youth 
programs  into  life-changing  experiences  for 
underserved  youth  in  your  community. 

Improve  your  existing  program  or  to  start  a  new  one 
through  a  deliberate  design  process  featuring  videos, 
information,  documents,  reference  links  and  activities 
organized  around  a  working  model  of  the  Youth 
Exploring  Science  fYES)  program  at  the  Saint  Louis 
Science  Center 

Get  started  by  touring  the  site! 
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Findings 

Using  the  YES  Program  Logic  Model  (Figure  2,  page  7)  as  a  guide,  quantitative  data 
are  grouped  in  the  By  The  Numbers  section  by  demographic  data,  psychometric  data, 
program  outputs,  short-term  impacts,  long-term  impacts,  and  strategic  impacts.  These 
are  followed  by  a  closer  look  at  qualitative  analysis  results  for  Attitudes  toward  Science 
and  Scientists  and  results  on  Career  Choices. 

By  the  Numbers  -  Summary  Data 

The  data  described  in  this  section  were  based  on  the  438  teens  participating  the 
YES  Program  over  the  past  three  years,  since  funding  from  the  Office  of  Naval 
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Research  began.  We  defined  a  participating  teen  as  one  who  attended  at  least  two 
days  in  any  of  the  eight  semesters  since  the  beginning  of  the  ONR  project  (Spring, 
Summer,  and  Fall  201 1  and  2012  plus  Spring  and  Summer  2013).  Throughout  this 
report,  unless  stated  otherwise,  we  used  438  as  the  number  of  teens  (i.e.  N  =  438)  in  all 
tables  and  graphs.  It  should  be  noted  that  after  an  initial  semester  of  participation,  some 
teens  may  not  have  participated  for  a  semester  or  two  to  work  another  job  or  participate 
in  other  activities,  and  then  returned  to  the  program  at  a  later  time.  Unless  a  teen 
officially  withdrew  from  the  YES  Program,  staff  members  kept  everyone  as  part  of  their 
database  in  order  to  communicate  and  maintain  relationships.  Relationship  was  a  key 
element  of  the  YES  Program.  This  means  that  there  was  a  difference  between  the 
number  of  “all  teens  in  the  program”  at  any  given  time,  and  “participating  teens.” 

Demographic  Data 

Demographic  Data  are  presented  in  the  following  figures  and  tables  to  describe  the 
YES  Teens  in  the  program.  While  there  is  a  balance  between  males  and  females  in  the 
program,  ethnicity  remains  predominately  African  American.  Because  data  span  three 
years,  grade  level  data  are  indicated  by  high  school  graduating  class. 

Gender  Data 


Table  2.  Gender  of  Participating  Teens 


Gender 

Total 

Female 

234 

Male 

204 

Total 

438 

Figure  3.  Gender  of  Participating  YES  Teens 
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Ethnicity  Data 


Table  3.  Ethnicity  of  Participating  Teens 


Ethnicity 

Total 

Black  or  African 

American 

382 

White 

10 

Asian 

8 

Hispanic 

4 

Multiple  Races 

1 

Declined  to 

Provide  Data 

33 

Total 

438 

Figure  4.  Ethnicity  of  YES  Teens 

It  should  be  noted  that  ethnicity  data  are  collected  by  YES  staff  from  the  teens,  some 
of  whom  decline  to  provide  that  information.  Traditionally,  teens  that  were  recent 
immigrants  self-reported  a  variety  of  very  specific  ethnic  groups.  To  keep  the  data  in 
Figure  4  simple  and  easy  to  read,  evaluators  grouped  teen  self-report  data  into  the  US 
Census  categories. 

Grade  Level  Data 

Youth  can  join  YES  as  early  as  age  fourteen,  which  means  YES  Teens  can  be  in 
grades  6-12.  Figure  5  and  Table  4  show  the  number  of  teens  participating  by  grade 
level  over  the  past  three  years.  Numbers  in  cohort  groups  and  grade  levels  do  not 
remain  consistent  for  year-to-year  comparisons  such  that  the  number  of  freshmen  in 
201 1  are  not  necessarily  the  sophomores  in  2012.  Incoming  New  Teens  may  be  in  any 
grade  level.  Additionally,  teens  occasionally  have  another  job,  family  issue,  or  school 
activity  that  prevents  participation  in  a  YES  Learning  Lab  during  one  year  even  though 
they  return  to  the  program  to  participate  in  the  next. 


Klein  Consulting 


17 


A110 


90 

80 

70 

8  60 

Q 

O 

E  50 

I  40 

E 

Z  30 
20 
10 
0 


62 


36 


1 


31 


79 


79 


73 


46 


J 


50 


68 


72 


59 


45- 


-54- 


51 


39  U  2010-2011 
“  2011-2012 
u 2012-2013 


Middle  School  Freshman  Sophomore  Junior  Senior 

Grade  Level 


Figure  5.  Grade  Levels  of  YES  Teens  in  2010-2011  (N  =  300),  2011-2012  (N  =  290), 
and  2012-2013  (N=256) 


Table  4.  Grade  Levels  of  YES  Teens  by  Year  of  Project 


2010-2011 

2011-2012 

2012-2013 

Middle  School 

62 

36 

31 

Freshman 

73 

79 

46 

Sophomore 

50 

79 

68 

Junior 

59 

45 

72 

Senior 

56 

51 

39 

Total  (N) 

300 

290 

256 

Cohort  Data 

Figure  6  tells  us  that  the  group  of  teens  entering  the  program  in  201 1  (Cohort  201 1 ), 
when  funding  from  the  Office  of  Naval  Research  was  received,  is  the  largest  group 
represented  in  YES.  SLSC  leadership  chose  to  recruit  smaller  numbers  of  incoming 
teens  in  the  springs  of  2012  and  2013.  Totals  include  only  those  teens  that  participated 
more  than  one  day,  and  not  teens  that  joined  the  cohort  and  left  after  their  first  Learning 
Lab. 
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Figure  6.  Number  of  Participating  YES  Teens  by  Cohort 


School  Data 


YES  Teens  attended  a  wide  variety  of  schools  -  public,  private,  homeschool,  and 
others,  in  Missouri  and  Illinois.  Since  some  of  the  teens  changed  schools  frequently, 
data  in  Figure  7  were  based  on  the  most  recent  high  school  or  middle  school  attended 
Of  the  438  participants,  only  415  provided  information  about  the  school  they  attended 
on  their  program  application  or  on  questionnaires. 


While  YES  Teens  attended  a  wide  variety  of  schools,  the  program  drew  substantially 
from  several  public  school  districts  in  the  St.  Louis  area.  Table  5  shows  the  six  school 
districts  from  which  over  50%  of  YES  participants  came.  These  percentages  were 
based  on  the  total  number  of  participants. 
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Public,  Missouri 
Private,  Roman  Catholic 
Public,  Missouri,  Charter 
Public,  Illinois 
Private,  Sectarian 
Homeschool 
Private,  Non-Sectarian 
Unknown 


Number  of  Participating  Teens  (N=438) 


Figure  7.  Number  of  Participating  YES  Teens  by  School  Type 
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Table  5.  Six  School  Districts  Totaling  over  50%  of  Participating  YES  Teens  ( N  =  438) 


School  District 

Percent 

St.  Louis  City 

31.1% 

Hazelwood 

7.5% 

Ferguson-Florissant  R-ll 

4.8% 

Normandy 

3.9% 

Parkway  C-2 

3.7% 

Ritenour 

3.0% 

Psychometric  Data 

Psychometric  data  tell  us  about  YES  Teens’  motivations  and  attitudes.  Why  did  they 
join  the  YES  Program,  and  why  do  they  keep  coming  back  each  semester?  Findings 
regarding  the  teens’  attitudes  toward  science  and  scientists  are  covered  in  a  separate 
section. 

Data  on  Why  Teens  Join  YES 

As  109  New  Teens  entered  the  program  in  Spring  201 1  (as  the  ONR  funding 
enabled  the  SLSC  to  bring  in  a  larger  pool  of  participants),  they  were  asked  why  they 
joined  the  YES  Program.  Ninety-five  responded  to  the  survey. 

The  survey  asked  New  Teens:  What  is  your  main  reason  for  wanting  to  join  the  YES 
Program?  They  were  asked  to  check  one  of  the  following,  or  to  write  in  something  under 
“other”. 

□  My  family  wanted  me  to 

□  My  mentors  wanted  me  to 

□  I  love  science 

□  I  needed  a  job 

□  The  money 

At  the  time  the  teens  joined  YES,  the  graduating  seniors  were  given  a  laptop.  This 
practice  ended  with  the  Interim  President  and  financial  restructuring  of  SLSC,  before 
most  of  the  respondents  graduated. 

On  the  survey,  many  teens  selected  more  than  one  response.  Since  it  was  difficult  to 
tell  which  one  category  was  most  important,  all  responses  were  counted.  Data 
collected  from  the  New  Teens  are  summarized  in  Figure  8. 


□  The  laptop 

□  I  wanted  work  experience 

□  My  friends  said  it  was  fun 

□  Other 
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Figure  8.  New  Teens’  reasons  for  joining  YES 


The  three  teens  checking  “other”  had  these  reasons  (in  their  own  words): 

•  For  future  career 

•  I  felt  like  it  was  an  opportunity  to  better  myself 

•  It  was  something  I  never  heard  of  and  wanted  to  try 


Data  on  Why  Teens  Keep  Coming  Back  to  YES 


In  the  spring  of  2013,  returning  YES  Teens  were  asked  why  they  kept  coming  back 
to  the  YES  Program  (N=122).  Again,  they  were  asked  to  check  one  of  the  following 
options,  but  many  selected  several.  Responses  are  summarized  in  Figure  9. 


□  My  family  makes  me 

□  I  love  science 

□  I  need  a  job/money 


□  I  like  the  work  experience 

□  I  like  teaching  the  kids 

□  Other 
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Figure  9.  YES  Teens’  reasons  for  coming  to  return  to  YES 

The  four  teens  checking  “other”  and  three  who  added  comments  in  the  same 
space  said  the  following  (in  their  own  words): 

•  family  tradition 

•  I  like  the  people 

•  I  love  being  around  my  friends  and  Matt 

•  I  make  myself  come 

•  some  supervisors 

•  something  to  do  on  the  weekends  (and  summer)  so  I  don't  have  to  lay 
around  the  house 

•  The  motivation  in  doing  what  I  want  to  do  -  preparing  me  for  the  real  life  in 
the  future. 

Data  indicate  the  work  experience  and  pay  bring  teens  into  the  program  and 
keep  them  coming  back. 

Program  Outputs 

Each  year,  sometimes  each  semester,  YES  educators  offer  different  STEM 
components  based  on  a  variety  of  factors,  including  interests  of  staff  and 
available  funding  (e.g.  specific  grants).  Table  6  provides  a  list  of  components  and 
the  number  of  participating  teens  per  component.  (Participating  teens  are  defined 
as  those  who  attended  more  than  one  day  in  that  semester.) 
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Table  6.  Components  Offered  by  Semester  with  Teen  Participants 


Spring 

2011 

Summer 

2011 

Fall 

2011 

Spring 

2012 

Summer 

2012 

Fall 

2012 

Spring 

2013 

Summer 

2013 

School  Year  Component  Groups 

New  Teens 

105 

- 

- 

61 

- 

- 

48 

- 

Agriscience 

- 

21 

34 

35 

24 

- 

- 

14 

Astronomy 

- 

- 

28 

34 

23 

- 

- 

11 

Atmospheric  Science 

- 

- 

- 

- 

- 

29 

25 

- 

Biofuels  &  Energy 

- 

- 

22 

35 

28 

35 

23 

- 

C3  (Climate  Change) 

14 

- 

- 

- 

- 

- 

- 

- 

Design  IT 

18 

- 

- 

- 

- 

32 

28 

10 

Health 

10 

- 

- 

- 

- 

- 

- 

- 

HOSCO 

- 

- 

- 

- 

- 

- 

- 

16 

Learning  Places 

21 

- 

- 

- 

- 

- 

- 

- 

Neuroscience 

- 

- 

24 

3 

- 

31 

35 

- 

Plant  Biochemistry 

- 

- 

25 

- 

- 

- 

- 

- 

Robotics 

18 

- 

38 

41 

22 

- 

- 

- 

SciJourn 

8 

13 

6 

8 

11 

- 

- 

- 

Science  Corner 

25 

- 

- 

- 

- 

- 

- 

- 

Sea  Perch 

18 

- 

- 

- 

- 

- 

- 

- 

Mystery  of  Matter 

- 

9 

- 

- 

8 

32 

31 

16 

Summer  Component  Groups 

Summertime  Science 

- 

96 

- 

- 

60 

- 

- 

60 

Exhibit  Lab 

- 

- 

- 

- 

71 

- 

16 

10 

Main  Building 

- 

67 

- 

- 

- 

- 

- 

- 

Offsite 

- 

60 

- 

- 

5 

- 

- 

53 

Science  on  the  Go 

- 

10 

- 

- 

- 

- 

- 

- 

Science  of  Learning 

- 

- 

- 

- 

- 

- 

- 

15 

Total 

237 

276 

177 

217 

252 

159 

206 

205 

Two  components  were  funded  through  the  National  Science  Foundation 
(NSF).  SciJourn  received  NSF  funding  through  the  University  of  Missouri-St. 
Louis  and  served  as  a  separate  component  in  the  school  year  and  supported  all 
components  in  the  summer.  Mystery  of  Matter  received  NSF  funding  through 
AAAS,  and  supported  the  other  components  during  the  2011-2012  academic 
year,  though  was  a  separate  component  in  other  sessions. 

Table  7  combines  the  figures  above  into  categories  based  on  the  STEM 
content. 
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Table  7.  STEM  Content  Offered  by  Semester  with  Teen  Participants 


Spring 

2011 

Summer 

2011 

Fall 

2011 

Spring 

2012 

Summer 

2012 

Fall 

2012 

Spring 

2013 

Summer 

2013 

New  Teens 

105 

- 

- 

61 

- 

- 

48 

- 

Astronomy 

- 

- 

28 

34 

23 

- 

- 

ii 

Biology/Env  Science 

49 

21 

105 

73 

52 

95 

83 

30 

Chemistry 

- 

9 

- 

- 

8 

32 

31 

16 

Engineering  &  Design 

75 

- 

38 

41 

93 

32 

44 

20 

Journalism 

8 

13 

6 

8 

11 

- 

- 

- 

Teaching  Science 

- 

233 

- 

- 

65 

- 

- 

128 

Total 

237 

276 

177 

217 

252 

159 

206 

205 

Attendance  remained  a  concern  of  the  evaluation  team  throughout  the 
project.  In  the  early  years  of  the  YES  Program,  as  part  of  the  Deliberate  Design, 
educators  regularly  contacted  teens  that  were  absent  to  be  sure  they  were  well 
and  were  coming  back.  In  recent  years,  many  educators  did  not  follow  the  same 
protocol.  Based  on  interview  data  and  staff  meeting  observations,  this  appeared 
to  be  connected  to  changes  in  professional  development  and  management 
practices  that  did  not  set  expectations  for  ongoing  contacts,  and  to  lack  of 
monitoring  of  staff  members  to  hold  them  accountable  for  this  responsibility.  After 
this  issue  was  brought  out  in  the  formative  evaluation  (year  2),  the  administrators 
began  to  address  the  issue. 

Table  8  provides  an  overview  of  YES  attendance  since  the  evaluation  began 
in  Spring  201 1 .  Row  1  shows  the  number  of  Learning  Lab  opportunities  (number 
of  days  sessions  were  held  in  each  semester).  Row  2  shows  the  total  number  of 
participating  teens  in  each  of  these  semesters,  and  row  3  shows  the  percentage 
of  attendance  for  each  semester.  Percent  attendance  was  calculated  using  the 
sum  of  the  number  of  teens  in  all  sessions  that  semester  divided  by  the  total 
participating  teens  times  number  of  days. 


Table  8.  Learning  Lab  Opportunities  in  and  Percent  Attendance  from  Spring  2010 
through  Summer  201 3 


Semester 

Spring 

2011 

Summer 

2011 

Fall 

2011 

Spring 

2012 

Summer 

2012 

Fall 

2012 

Spring 

2013 

Summer 

2013 

Days 

15 

36 

11 

14 

32 

11 

17 

19 

Participating 

Teens 

237 

278 

177 

217 

252 

159 

206 

205 

Percent 

Attendance 

67.9% 

82.4% 

71.1% 

52.6% 

83.4% 

70.8% 

60.4% 

78.6% 

Attendance  in  most  semesters  is  affected  by  teens  participating  in  sports, 
school  and  community-based  extracurricular  activities  and  family  emergencies. 
The  lower  level  of  attendance  in  Spring  2012  coincided  with  layoffs  of  staff. 
Teens  form  relationships  with  individual  staff  members,  and  as  evident  in  focus 
groups  and  interviews  with  teens,  some  had  strong  feelings  when  their 
component  leader  lost  her/his  job.  When  two  components  were  cancelled,  only 
two  teens  immediately  left  the  program;  however,  lower  levels  of  attendance 
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throughout  the  semester  and  conversations  with  teens  indicated  that  budget  cuts 
with  associated  staff  layoffs  disrupted  the  staff-teen  relationships  which  underlie 
regular  program  attendance. 

Short-term  Impacts 

Several  stakeholders  focus  on  high  school  graduation  rates  and  post¬ 
secondary  plans  as  indicators  of  success.  They  asked  for  rates  of  YES  Teens 
compared  with  other  teens  in  the  region.  However,  what  appears  to  be  a  simple 
calculation  and  comparison  is  actually  full  of  challenges. 

High  School  Graduation  Data 

When  calculating  the  high  school  graduation  rates,  the  following  challenges 
arose. 

•  YES  Teens  attend  a  wide  variety  of  private  schools  and  public  school 
districts  in  Illinois  and  Missouri. 

•  Complete  high  school  graduation  data  from  area  school  districts  in 
Missouri  and  Illinois  were  only  available  for  201 1  and  earlier. 

•  YES  Program  staff  members  have  only  collected  data  from  seniors  in 
2012  and  2013,  not  2011. 

•  Data  are  missing  from  YES  2012  and  2013  seniors  that  staff  members 
were  unable  to  reach. 

•  Names  and  numbers  of  seniors  differed  between  those  kept  by  the  YES 
staff  and  those  in  the  evaluators’  database,  which  were  based  on  survey 
data  and  program  records. 

To  address  the  challenges  and  still  arrive  at  comparison  numbers,  the 
following  assumptions  were  made. 

•  Graduation  rates  for  school  districts  can  be  weighted  based  on  the  percent 
of  YES  seniors  attending  each  district. 

•  School  district  rates  are  stable  enough  to  compare  201 1  rates  with  YES 
graduation  rates  from  2012  and  2013. 

•  Differences  between  YES  staff  data  and  the  evaluation  database  were 
minimal,  such  that  high  school  graduation  data  could  be  used  from  YES 
staff,  and  school  district  data  could  be  used  from  the  evaluation  database 
for  comparison  rates. 

•  To  address  missing  YES  senior  data,  including  only  seniors  contacted  (i.e. 
those  not  reached  are  excluded)  can  provide  an  accurate  estimate  for 
comparison. 

Weighted  rates  for  comparison  were  calculated  much  like  weighted  grades  in 
a  high  school  class. 

1 .  Number  of  YES  seniors  per  public  school  district  or  private  school  were 
obtained  from  the  evaluation  database 

2.  High  school  graduation  rates  from  public  school  districts  and  private 
schools  were  obtained  from  the  Missouri  and  Illinois  departments  of 
education. 
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3.  Percent  of  YES  seniors  attending  each  district  or  school  were 
calculated 

4.  Results  from  2  and  3  above  were  multiplied  to  create  a  weighted 
percent  for  each  district/school 

5.  The  total  from  4  above  was  calculated  to  give  the  comparison  rate  - 
one  based  on  seniors  from  2012  and  a  second  from  2013  seniors 


Table  9.  Comparisons  of  high  school  graduation  rates 


Groups 

2012 

2013 

YES  Seniors  (missing  data  excluded) 

96.36 

100.00 

Area  school  weighted  comparison 

75.43 

78.42 

Missouri  state  average* 

82.40 

83.10 

Illinois  state  average* 

80.40 

77.70 

*  State  averages  were  obtained  from  www.americashealthrankings.org/ALL/Graduation/ 

From  the  results  in  Table  9  and  Figure  10,  it  is  clear  that  YES  Teens  who  are 
seniors  in  high  school  graduate  at  a  rate  higher  than  their  peers  from  the  same 
area  schools  and  from  the  states  of  Missouri  and  Illinois.  Visually,  in  looking  at 
Figure  10,  the  difference  between  the  comparison  (Area  Students)  and  the  YES 
seniors  when  excluding  unknowns  from  the  calculations  (YES  Teens)  is 
impressive. 


2013 
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Illinois 

* 


Missouri 
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1  1  1  1 
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Area  Students  78.42 

YES  Teens 

100.00 

Area  Students  75.43 

YES  Teens 

96.36 

0  10  20  30  40  50  60  70  80  90  100 

High  School  Graduation  Rate 


Figure  10.  2012  and  2013  High  School  Graduation  Rate  Comparisons 

By  making  assumptions  in  the  calculations  to  address  the  challenges  faced 
by  missing  data  from  the  YES  Program  and  the  states,  the  comparison  of  high 
school  graduation  rates  shows  that  YES  Teens  who  remain  in  the  program  until 
their  senior  year  in  high  school  graduate  at  a  higher  rate  than  their  peers  in  the 
same  area  schools. 
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Critical  is  the  phrase  “YES  Teens  who  remain  in  the  program  until  their  senior 
year  in  high  school.”  School  district  and  state  graduation  rates  must  include 
students  who  began  as  freshmen  but  dropped  out  of  school  before  or  during  their 
senior  year  (and  did  not  transfer).  The  YES  Program  does  not  keep  attrition  data 
and  does  not  follow-up  with  teens  that  drop  out  of  the  program  to  determine 
whether  or  not  they  graduated  from  high  school.  Thus,  when  comparing  high 
school  graduation  rates  between  YES  seniors  and  weighted  averages  for  area 
schools  or  states,  the  differing  definitions  of  graduation  should  be  noted  (i.e.  the 
comparison  is  between  apples  and  oranges). 

Post-secondary  Plans 

In  addition  to  knowing  the  graduation  rates,  it  is  helpful  to  know  the  plans  of 
the  YES  Teens  for  their  post-secondary  education.  To  measure  short-term 
impacts,  surveys  of  participating  teens  included  questions  about  their  plans  for 
after  high  school  -  job,  college,  trade  school,  or  military. 

As  Figure  1 1  shows,  over  half  of  the  seniors  in  the  graduating  class  of  2012 
planned  to  attend  a  4-year  college.  Only  3%  (“other”)  had  not  yet  graduated  from 
high  school.  YES  staff  members  were  unable  to  reach  12%  (“unknown”)  of  the 
graduating  seniors  in  the  summer  of  2012. 


Figure  11.  Post-High  School  Plans  of  Seniors  in  May  2012 


In  2013,  YES  staff  members  were  unable  to  reach  nine  of  the  graduating 
seniors,  and  it  is  unknown  whether  or  not  these  teens  graduated  from  high 
school.  These  nine  teens  are  included  in  the  unknown  data  in  Figure  12.  All 
teens  contacted  had  graduated.  Figure  12  shows  that  46%  of  the  seniors  in  the 
graduating  class  of  2013  planned  to  attend  a  4-year  college.  Two  teens  had 
joined  the  military,  one  with  the  Navy  and  one  with  the  Army. 
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Figure  12.  Post-High  School  Plans  of  Seniors  in  May  2013 

Comparable  data  from  201 1  were  collected  by  program  staff,  however  are  no 
longer  available  due  to  staff  turnover.  Survey  data  collected  by  the  evaluation 
team  from  a  sample  of  the  graduating  class  of  201 1  in  Summer  201 1  indicated 
that  94%  of  respondents  applied  to  a  trade  school,  college,  or  university  (though 
2  had  not  heard  back  yet),  while  one  teen  joined  the  US  Navy  and  one  joined  the 
Job  Corps. 

Long-term  impacts 

With  teens  in  the  YES  Program  for  four  and  a  half  or  more  years,  long-term 
impacts  include  those  impacts  recognized  after  they  have  left  the  program  and 
gone  on  to  college,  the  military,  and/or  careers.  To  begin  to  measure  the  long¬ 
term  impact  of  the  program,  an  online  survey  was  developed  for  former  YES 
participants  asking  about  their  experiences  during  and  after  YES.  Twenty-six 
individuals  participated  in  the  survey  by  the  deadline,  with  22  surveys  completed 
by  former  YES  Teens  to  the  extent  that  they  could  be  included  in  the  results 
reported  here.  The  small  number  of  responses  (N=22)  can  give  a  limited  picture 
of  the  activities  of  YES  alumni/alumnae,  but  caution  should  be  exercised  in 
drawing  any  conclusions  about  program  impact. 

This  pilot  provided  insights  into  collecting  data  from  a  highly  mobile  population 
that  had  not  remained  in  contact  with  the  program.  As  a  small  pilot,  however,  it 
could  not  provide  evidence  of  the  impact  of  the  YES  Program  on  all  YES 
participants.  In  this  report,  we  can  only  provide  an  overview  of  the  data  collected 
and  the  lessons  learned  in  the  process. 
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Of  the  22  respondents,  six  did  not  provide  demographic  data.  From  data 
provided,  ten  were  female  and  six  were  male.  Fourteen  were  African  American, 
one  was  Caucasian,  and  one  was  multiracial.  One  was  a  naturalized  citizen 
while  the  rest  were  native  US  citizens.  Five  participated  in  the  program  less  than 
three  years,  with  the  majority  participating  four  or  more  years. 

Twenty  of  the  22  respondents  graduated  from  high  school  and  two  earned  a 
GED  (from  cohorts  2006  and  2009).  After  high  school,  18  of  the  respondents 
(82%)  attended  a  trade  school,  college  or  university  after  high  school.  Five  of 
those  had  graduated  from  college,  with  two  in  graduate  school  at  the  time  of  the 
pilot  survey.  Eleven  were  still  in  trade  school  (1  participant)  and  college  (10 
participants).  Areas  of  study  varied,  as  listed  below.  Business  and  engineering 
were  the  two  most  prominent  among  respondents. 

•  Biological  and  Biomedical  Sciences  =  1 

•  Business,  Management,  Marketing,  and  Related  Support  Services  =  4 

•  Cosmetology  =  1 

•  Education  and  English  =  1 

•  Engineering  =  3 

•  Fashion  Design  =  1 

•  Health  Professions  and  Related  Programs  =  1 

•  History  =  1 

•  Mathematics  and  Statistics  =  1 

•  Nursing  =  1 

•  Psychology  =  1 

•  Sociology  =  1 

•  Visual  and  Performing  Arts,  with  Education  =  1 

Of  the  22  respondents  included  in  this  analysis,  13  were  students  at  the  time 
of  the  survey.  Most  of  these  were  employed.  Table  10  provides  the  employment 
status  of  the  alumni/alumnae  in  spring  2013.  Table  11  provides  the  types  of 
organizations  for  which  they  worked.  All  data  were  self-report,  thus  some 
discrepancies  appeared. 


Table  10.  Employment  Status  of  Respondents 


Current  Employment  Status 

Number 

Full-time  Employee 

8 

Part-time  Employee 

5 

Unemployed  &  seeking  employment 

4 

Internship 

2 

Self-employed  and  holding  temp  position 

1 

Work-study 

1 

Full-time  student  seeking  part-time  job 

1 
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Table  1 1 .  Types  of  Organizations  Employing  Respondents 


Type  of  Organization 

Number 

Private  For-Profit  Company 

3 

International  Organization 

3 

K-12  School 

2 

State  or  Local  Government 

2 

Self-employed 

2 

Work-study 

2 

Unclear 

2 

Federal  Government 

1 

Unemployed 

5 

The  survey  provided  lists  of  categories  respondents  used  to  characterize  the 
focus  of  their  occupations  (Table  12)  and  current  positions  (Table  13). 


Table  12.  Occupations  of  Respondents 


Type  of  Occupation 

Number 

Sales  and  Related  Occupations 

3 

Business  and  Financial  Operations 

3 

Education,  Training,  and  Library 

2 

Management 

2 

Community  and  Social  Service 

1 

Food  Preparation  and  Serving 

1 

Arts,  Design,  Entertainment,  Sports,  and  Media 

1 

Architecture  and  Engineering 

1 

Healthcare  Support 

1 

Healthcare  Practitioners  and  Technical 

1 

Office  and  Administrative  Support 

1 

None  listed  or  unemployed 

5 

Table  13.  Current  Positions  of  Respondents 


Position  Categories 

Number 

Sales,  marketing,  advertising  or  public  relations  manager 

4 

Administrative  support,  clerical  worker,  secretary 

3 

Classroom  teacher  (K-12) 

2 

Engineer 

2 

Other  creative  profession 

2 

Human  resources  or  labor  relations  professional 

1 

Manager,  administrator,  or  management  consultant 

1 

Visual  artist  or  designer 

1 

Salesperson,  broker,  or  agent 

1 

Medicine  -  other  health  services  professional 

1 

No  current  position 

4 
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Of  the  four  unemployed  respondents,  one  was  seeking  a  position  as  an 
educator  or  in  a  managerial  role.  One  was  seeking  a  position  as  a  nurse. 
Another  was  looking  for  managerial  or  financial  positions.  The  fourth  did  not 
specify  a  type  of  position. 


The  survey  asked  about  the  types  of  knowledge  and  skills  required  by  their 
positions  (See  Figure  13).  Most  required  clear  communication,  collaboration, 
creative  thinking,  accessing  and  evaluating  information,  and  using  and  managing 
information.  While  math  skills  were  required  by  just  over  half  of  the  respondents’ 
positions,  science  and  engineering  were  required  by  less  than  half. 
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Figure  13.  Average  Rating  of  Knowledge  Involved  in  Respondents’  Current  Positions 
(N=19) 

When  asked  what  influenced  their  choice  of  career,  family  and  school 
experiences  were  most  prominent  with  YES  experiences  a  close  third 
(Figure  14). 
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Figure  14.  Average  Rating  of  Influences  on  Respondents’  Career  Choices  (N=18) 


When  asked  how  well  the  YES  Program  prepared  them  for  post-secondary 
education  respondents  gave  a  3.80  average  on  a  5-point  scale  with  5  as  greatly 
helpful.  When  asked  how  well  the  program  prepared  them  for  the  workforce,  the 
resulting  average  was  4.53  on  the  same  5-point  scale.  When  asked  how 
influential  the  YES  Program  was  on  college  and  career  choices,  the  average  was 
3.88. 

Results  indicated  that  networking 
teambuilding,  and  teaching  younger 
children  were  the  aspects  of  the 
program  that  influenced  respondents 
the  most.  When  asked  a  slightly 
different  question,  “To  what  extent 
did  these  elements  of  the  YES 
Program  positively  impact  you?” 
rated  from  1  (no  positive  impact)  to  5 
(high  positive  impact),  teaching 
younger  children  and  developing  job 
skills  received  the  highest  averages, 
followed  by  earning  an  income  and 
public  speaking  experiences. 

When  asked  about  the  influence  of  the  YES  Program  on  their  attitudes, 
results  showed  that  the  greatest  influences  were  on  attitudes  toward  adults, 
learning,  young  children,  workplace  policies  and  procedures,  and  teachers.  “On  a 
scale  of  1  -  5,  please  rate  to  what  extent  the  YES  Program  influenced  your 
attitudes  toward  the  following?  1  =  have  a  more  negative  attitude  to  5  =  have  a 
more  positive  attitude.”  Respondents  were  given  the  option  to  mark  “not 
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applicable”.  Results  are  summarized  in  Table  14,  from  most  positive  influence  to 
least  positive. 


Table  14.  Influence  of  YES  Program  on  Attitudes 


Attitude  Toward: 

N 

Average 

Adults 

16 

4.40 

Learning 

16 

4.38 

Young  Children 

16 

4.31 

Workplace  Policies  and  Procedures 

16 

4.19 

Teachers 

16 

4.19 

Other  Professionals 

15 

4.13 

Administrators  (i.e.  Bosses)  in  Any  Work  Setting 

16 

4.06 

Science  in  General 

16 

4.00 

Conducting  Science  Investigations 

15 

3.93 

College 

15 

3.93 

Professional  Scientists 

15 

3.87 

Formal  Education  (High  School  and  College) 

15 

3.73 

Informal  Education  (Museums,  After  School 

Programs,  Educational  TV,  Etc.) 

15 

3.73 

Recycling 

16 

3.88 

Composting 

14 

3.71 

Global  Warming  and  Climate  Change 

14 

3.64 

Space  Exploration 

14 

3.64 

Cloning 

14 

3.36 

The  pilot  survey  provided  direct  quotes  from  the  respondents;  a  couple  of 
these  related  to  impact  on  career  and  STEM  follow. 


YES  taught  me  how  to  be  a  key  role  player  in  the  work 
environment,  it  also  taught  me  to  never  settle,  there's  always  room 
for  improvement  in  the  work  area  and  I  can  go  as  far  as  my 
dreams.  (2008-2012,  male) 

I  am  forever  grateful  to  the  YES  program.  I  love  the  program.  The 
YES  program  influenced  the  work  that  I  put  forth  in  other  positions 
I've  held  since  leaving  the  program.  It  taught  me  work  ethic  and 
organizational  skills  that  I'll  never  forget. . ..  My  participation  in  the 
YES  program  helped  me  stay  in  the  sciences.  It  kept  me  motivated 
and  kept  me  loving  science  and  mathematics.  (1999-2003,  female) 

Information  collected  from  the  22  former  YES  Teens  in  this  pilot  online  survey 
provided  evidence  of  the  type  of  impact  data  that  could  be  collected  and  insights 
into  how  to  improve  data  collection,  with  additional  investment  to  provide 
incentive  for  response  and  identification  of  up-to-date  contact  information.  A 
separate  report  (Klein  &  Tisdal,  2013)  included  a  full  discussion  of 
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recommendations  for  future  surveys. 

Since  YES  Program  staff  do  not  maintain  data  on  all  former  YES  Teens,  it  is 
impossible  to  know  if  the  22  responses  to  this  pilot  survey  were  representative  of 
the  YES  alumni/alumnae  population.  An  additional  challenge  was  the  large 
number  of  survey  respondents  still  in  school. 

Recommendations  for  future  work  included: 

•  Changes  to  the  survey  (detailed  in  the  separate  report) 

•  Creation  of  an  ongoing  data  collection  and  tracking  system  for 
alumni/alumnae 

•  Incentives  for  survey  completion 

•  Addition  of  open-ended  questions  on  the  survey  or  in  follow-up  interviews 

•  Surveys  for  teens  who  dropped  out  of  the  YES  Program  prior  to  high 
school  graduation 

Strategic  Impacts 

The  Project  Objectives  state  that  project  leaders  will: 

1 .  Create  a  strategic  plan  for  national  expansion  of  the  SLSC  Community 
STEM  Outreach  Program 

2.  Identify  resources,  including  national  partners,  for  national  outreach 

Toward  these  objectives,  under  the  leadership  of  Diane  Miller,  SLSC  identified 
national  science  center  and  museum  partners  with  existing  youth  programs  or 
plans  for  such  programs.  Representatives  from  these  nine  national  partners 
joined  SLSC  staff,  representatives  from  the  ONR,  and  local  partners  on  April  21- 
23,  2013  in  St.  Louis  to  discuss  the  YES  Program  model  and  its  potential 
dissemination.  Organizations  represented  included: 

•  The  Office  of  Naval  Research 

•  Bishop  Museum 

•  California  Science  Center 

•  Great  Lakes  Science  Center 

•  Lawrence  Hall  of  Science 

•  Museum  of  Life  and  Science 

•  National  Museum  of  American  History,  Smithsonian  Institution 

•  New  Mexico  Museum  of  Natural  History  and  Science 

•  Reuben  Fleet  Science  Center 

•  Science  Museum  of  Minnesota 

•  Saint  Louis  Science  Center 

•  Missouri  Botanical  Gardens 

•  Saint  Louis  Zoo 

•  Daugherty  Group 

•  Hosco  Farms 

•  Klein  Consulting 

•  Tisdal  Consulting 
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On  the  second  day  of  the  meeting,  three  breakout  groups  met  to  begin 
discussing  possible  areas  for  collaboration:  Science  Learning,  Youth 
Development,  and  Workforce  Development.  A  conceptual  model  (Figure  15)  was 
discussed,  and  the  focus  of  discussion  shifted  to  ways  to  bring  the  three  together 
-  to  increase  the  “sweet  spot”  at  the  center. 


Figure  15.  Conceptual  Model 

While  there  was  no  consensus,  the  group  agreed  to  carry  on  the  conversation 
through  a  Thinkfinity  group  discussion,  to  be  organized  by  one  volunteer  from  a 
national  partner.  The  next  step  would  be  for  SLSC  leaders  to  develop  action 
items  and  solicit  feedback.  After  the  April  meeting,  the  Thinkfinity  site  was 
established  but  few  meeting  participants  joined  or  commented.  With  lack  of 
follow-up  discussion  from  conference  attendees  and  guidance  from  ONR 
regarding  limited  funding  opportunities,  the  SLSC  administration  concluded  that 
an  appropriate  role  for  the  institution  would  be  to  create  a  tool  for  national 
dissemination  (the  Circles  of  Support  website),  rather  than  take  a  leadership  role 
in  obtaining  federal  funding  or  working  toward  consensus  on  a  project  with  the 
national  partners. 


Attitudes  Toward  Science  and  Scientists 

To  measure  changes  in  attitudes  toward  science  and  scientists,  evaluators 
used  a  modified  version  of  the  Test  of  Science-Related  Attitudes,  (TOSRA2).  The 
TOSRA,  designed  by  Fraser  (1981)  to  measure  secondary  science  students’ 
attitudes  toward  science,  consisted  of  70  statements  with  seven  subscales  using 
a  5-point  Likert  Scale  (strongly  agree,  agree,  not  sure,  disagree,  and  strongly 
disagree).  TOSRA  has  been  used  with  youth  around  the  world,  and  has  been 
shown  to  be  valid  and  reliable  for  American  teens.  A  modified  version  (TOSRA2) 
developed  by  Ledbetter  and  Nix  (2002)  was  used  in  this  study,  consisting  of  35 
pre-test  items  and  35  post-test  items  with  negatively  and  positively  phrased  items 
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balanced  on  each  test.  Results  are  summarized  here.  Detailed  results  of  this  test 
can  be  found  in  Appendix  D. 

The  seven  subscales  were  as  follows: 

•  Social  Implications  of  Science  (S)  -  Do  youth  recognize  the  benefits  and 
drawbacks  of  scientific  advances  to  society? 

•  Normality  of  Scientists  (N)  -  Do  youth  see  scientists  as  real  people  rather 
than  media-produced  stereotypes? 

•  Attitude  toward  Scientific  Inquiry  (I)  -  Do  youth  view  experimentation  and 
inquiry  as  a  way  to  gain  understanding  of  the  natural  world? 

•  Adoption  of  Scientific  Attitudes  (A)  -  Have  youth  adopted  the  attitudes  of 
scientists,  such  as  open-mindedness  and  self-assessment? 

•  Enjoyment  of  Science  Lessons  (E)  -  To  what  degree  do  youth  enjoy  their 
lessons  in  school  science  classes? 

•  Leisure  Interest  in  Science  (L)  -  To  what  degree  are  youth  interested  in 
science  out  of  school,  and  outside  of  the  YES  Program? 

•  Career  Interest  in  Science  (C)  -  Do  youth  have  an  interest  in  pursuing  a 
science  related  career? 

The  TOSRA2  was  administered  in  YES  Learning  Labs  on  four  different 
occasions  as  summarized  in  Table  15.  Grade  level  in  school  was  provided  by 
most  teens  on  the  test;  however,  grade  level  could  not  be  found  in  the  YES 
staff’s  database  for  two  of  the  teens  (one  joined  YES  in  201 1 ;  the  other  in  2012). 
For  summer  tests,  grade  level  was  defined  as  the  level  the  YES  Teen  would 
enter  in  the  fall. 


Table  15.  Count  of  TOSRA2  respondents  in  each  grade  level. 


Unknown 

7-8tn 

9™ 

10tn 

Hth 

12tn 

Grad 

Total 

Pre-test,  March  2012 

0 

21 

47 

34 

21 

10 

0 

133 

Post-test,  July  2012 

1 

3 

18 

63 

54 

29 

27 

195 

Pre-test,  April  2013 

1 

14 

21 

30 

24 

20 

0 

110 

Post-test,  July  2013 

0 

2 

18 

29 

43 

15 

8 

115 

The  matched  pairs  used  for  this  analysis  were  scores  from  teens  taking  both 
the  2012  spring  pre-test  and  the  2013  summer  post-test,  which  provided  the 
greatest  time  between  tests.  Table  16  summarizes  the  teens  included  in  the 
matched  pairs  by  grade  level  in  school. 

Table  16.  Count  of  teens  taking  the  spring  2012  pre-test  AND  the  summer  2013  post¬ 
test  by  grade  level  at  post-test. 


9™ 

10tn 

Hth 

12th 

Grad 

Total 

Matched  Pairs 

1 

8 

26 

7 

2 

44 

Included  in  the  matched  pairs  were  24  males  and  20  females.  Data  included 
39  African  American  teens,  two  Asian  teens  who  were  recent  immigrants  from 
Nepal,  one  Caucasian,  one  Hispanic,  and  one  teen  for  whom  no  ethnicity  data 
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were  available.  In  addition  to  these  variables  and  teens’  grade  levels,  we 
included  length  of  time  (in  months)  that  respondents  had  been  in  the  program  at 
the  post-test  date  (Table  17). 

Table  17.  Count  of  teens  taking  the  spring  2012  pre-test  AND  the  summer  2013  post¬ 
test  by  length  of  time  in  the  YES  Program  at  post-test  in  months. 


Months  in  YES  at  Post-test 

Frequency 

17 

23 

29 

18 

41 

2 

53 

1 

The  findings  include  only  those  results  from  the  matched  pairs.  Ethnicity  was 
not  used  in  the  analysis  due  to  the  small  numbers  of  non-African  American  teens 
in  the  sample.  Analysis  focused  on  gender,  grade  level  as  an  indication  of  age 
and  maturity,  and  length  of  time  in  the  program.  Length  of  time  was  not  an 
indication  of  hours  of  participation,  but  rather  the  number  of  months  a  teen  was 
associated  with  the  program. 

Matched  Pair  Results 

Analysis  of  the  matched  pairs  found  that  scores  increased  from  pre  to  post¬ 
test  in  all  but  two  categories  (Attitude  Toward  Inquiry  and  Career  Interest). 
Correlation  coefficients  were  found  to  be  significant  in  six  categories  and  on  the 
total  scores.  Two-sided  paired  t-tests  found  significant  differences  (p  <  0.05)  in 
five  of  the  categories  and  the  total  scores.  Table  18  gives  means  for  each 
category  and  the  total  score  with  results  from  the  analyses. 


Table  18.  TOSRA2  Pre-  and  Post-test  means  by  TOSRA  category 


Pre- 

Test 

Post- 

Test 

Mean 

Difference 

Correlation 

Coefficient 

t 

(2-sided) 

Social  Implications 

17.05 

18.23 

-1.18 

0.383** 

-2.48* 

Normality 

15.59 

18.45 

-2.86 

0.460** 

-7.92** 

Attitude  toward 

Inquiry 

19.43 

18.16 

1.24 

0.352* 

2.42* 

Adoption  of  Attitudes 

17.73 

18.50 

-0.77 

0.290 

-1.77 

Enjoyment 

16.95 

18.14 

-1.18 

0.494** 

-2.11* 

Leisure  Interest 

14.55 

16.27 

-1.73 

0.535** 

-2.90** 

Career  Interest 

17.00 

16.16 

0.84 

0.500** 

1.43 

Total  Score 

118.30 

123.91 

-5.61 

0.649** 

-2.84** 

**  Significant  at  p<0.01 
*  Significant  at  p<0.05 


These  results  suggest  that  between  spring  2012  and  summer  2013,  respondents’ 
attitudes  changed  in  the  following  ways.  After  16  additional  months  in  the  YES 
Program  (i.e.,  16  months  between  testing),  youth  were: 

•  More  likely  to  recognize  the  benefits  and  drawbacks  of  scientific  advances 
to  society  --  Social  Implications  of  Science  (S) 
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•  More  likely  to  see  scientists  as  real  people  rather  than  media-produced 
stereotypes  -  Normality  of  Scientists  (N) 

•  Less  likely  to  view  experimentation  and  inquiry  as  a  way  to  gain 
understanding  of  the  natural  world  (gender  differences  are  described 
below)  -  Attitude  toward  Scientific  Inquiry  (I) 

•  Slightly  more  likely  to  have  adopted  the  attitudes  of  scientists,  though  the 
adoption  of  attitudes  was  relatively  weak  (i.e.  not  significantly  different)  - 
Adoption  of  Scientific  Attitudes  (A) 

•  More  likely  to  indicate  enjoyment  of  their  lessons  in  school  science 
classes  -  Enjoyment  of  Science  Lessons  (E) 

•  More  interested  in  science  out  of  school,  and  outside  of  the  YES  Program 
-  Leisure  Interest  in  Science  (L) 

•  Slightly  less  interested  in  pursuing  a  science  related  career  (gender 
differences  are  described  below)  -  Career  Interest  in  Science  (C) 

Overall  test  scores  showed  a  significant  increase  from  pre-test  to  post-test. 

Figure  16  provides  a  graphic  image  depicting  these  differences,  including  the 
negative  change  in  means  (i.e.  the  red  post-test  column  is  lower  than  the  blue 
pre-test  column)  for  Attitude  Toward  Inquiry  and  Career  Interest. 
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Figure  16.  Mean  scores  for  the  spring  2012  pre-test  and  summer  2013  post-test. 
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Results  of  the  matched  pair  analyses 
suggest  that  YES  Teens  changed  their 
attitudes  significantly  in  four  categories  and 
overall  scores  from  the  first  pre-test  given  in 
March  2012  to  the  post-test  given  16  months 
later  in  July  2013.  Attitudes  changed  in  the 
“positive”  direction  in  the  categories  of  Social 
Implications  of  Science,  Normality  of 
Scientists,  Enjoyment  of  Science  Lessons, 
and  Leisure  Interest  in  Science.  YES  Teens 
also  indicated  a  slight  positive  change  in  an 
Adoption  of  Scientific  Attitudes.  Attitudes 
changed  in  a  “negative”  direction  in  the  Attitude  toward  Scientific  Inquiry  and 
Career  Interests  in  Science  categories.  The  terms  positive  and  negative  refer  to 
whether  or  not  the  change  occurred  in  the  direction  anticipated  by  the  goals  of 
the  YES  Program  and  by  the  prevailing  views  of  what  scientifically  literate 
citizens  should  believe  about  science  and  scientists.  For  example,  one  view 
dominant  among  scientists  when  the  TOSRA  was  developed  was  that  it  is  better 
to  find  answers  through  your  own  experimentation  rather  than  through 
collaboration. 

Analyses  took  into  account  a  number  of  factors  that  could  affect  attitudes: 
gender,  ethnicity,  age/maturity,  and  time  in  the  YES  Program.  The  ethnic 
diversity  of  the  matched  pairs  was  similar  to  that  of  the  YES  Program,  and  did  not 
have  enough  variation  to  draw  conclusions  about  differences  in  attitudes  based 
on  ethnicity.  No  significant  differences  in  attitude  were  found  by  gender  on  gain 
scores,  though  males  tended  to  rate  items  lower  in  general  than  females  and 
females  rated  Attitude  Toward  Inquiry  significantly  higher  than  males  on  the  post¬ 
test.  A  significant  difference  was  found  between  length  of  time  in  the  program 
and  Attitude  toward  Inquiry  suggesting  the  longer  a  teen  was  in  the  YES 
Program,  the  less  likely  she  or  he  was  to  view  experimentation  and  inquiry  as  a 
way  to  gain  understanding  of  the  natural  world.  No  significant  differences  were 
found  based  on  age  as  an  indication  of  maturity,  though  the  9th  grader  rated 
categories  lower  on  the  post-test  and  the  two  high  school  graduates  rated  all 
except  two  categories  higher  on  the  post-test  than  on  the  pre-test. 

Findings  from  each  subscale  provide  more  detail  and  can  be  found  in 
Appendix  D. 

Career  Choices 

In  conjunction  with  the  TOSRA2  subscale  on  career  interests,  teens  were 
asked  about  career  choices  in  surveys  given  each  year.  It  should  be  noted  that 
middle  and  high  school  youth  vary  in  their  approaches  to  career  choices.  Some 
know  their  career  choice  from  an  early  age,  and  programs  like  the  YES  Program 
support  that  choice.  Some  don’t  know  what  career  they  want  to  pursue  and  may 
not  decide  until  college  or  later.  A  few  may  have  interests  and  find  that  OST 
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programs  give  them  the  information  and  career  exposure  they  need  to  decide  or 
narrow  their  options. 

The  surveys  asked  the  YES  Teens  to 
list  up  to  five  careers  they  were  interested 
in  pursuing.  Responses  varied  widely. 

Data  included  attendance,  years  in  the 
YES  Program,  grade  in  school,  and 
gender.  We  looked  for  trends  in 
responses.  Did  responses  get  more 
specific  over  time?  Did  teens  list  fewer 
choices  over  time,  indicating  that  they  had 
narrowed  their  choices?  Did  teens  list 
more  STEM  choices  over  time?  No 
significant  differences  were  found  for  any 
of  these  by  any  of  the  variables  analyzed 
(using  a  yC). 

Including  all  responses  by  teens  to  the  surveys  (N=294),  Figure  17  provides 
the  number  of  teens  who  listed  at  least  one  STEM,  Science,  Technology,  Math, 
Engineering,  and  Education  related  career  (columns).  Male  and  female 
responses  are  indicated  separately.  Thus,  out  of  294  responses,  244  (83%)  listed 
at  least  one  STEM  career  (135  females  and  109  males).  Most  of  those  were 
science  related  careers.  Since  the  YES  Program  engages  teens  in  teaching 
STEM  activities  to  younger  children,  career  interest  in  education  fields  was 
included  in  the  analysis,  resulting  in  55  teens  (39  female  and  16  male)  listing  a 
choice  in  teaching  or  early  childhood  careers. 
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Figure  1 7.  Career  Interests  by  Category  and  Gender 
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To  take  a  closer  look  at  gender  differences,  Figure  18  indicates  the  percent 
male  and  female  for  each  career  category. 
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Figure  18.  Career  Choice  by  Percent  Male  and  Female 

On  the  Career  Interest  subscale  of  the  TOSRA2  instrument,  teens’  scores 
differed  by  gender,  though  the  difference  was  not  significant.  It  is  interesting  to 
note  that  scores  decreased  for  both  male  and  female  teens  from  pre-test  to  post¬ 
test,  with  a  larger  decrease  among  males  (Table  19). 


Table  19.  Career  Interest  Scores  on  TOSRA2  by  Gender  for  Matched  Pairs 


Pre-Test  2012 

Post-Test  2013 

Gain 

Male 

17.29 

16.00 

-1.29 

Female 

16.65 

16.35 

-0.30 

In  addition  to  the  surveys  and  TOSRA,  teens  participated  in  focus  groups  and 
interactive  interviews.  Career  choices  were  a  topic,  though  not  all  teens  had 
decided  on  a  career.  Results  indicated  that  very  few  teens  changed  their  career 
interests  due  to  their  participation  in  the  YES  Program. 

From  the  surveys,  TOSRA2,  focus  groups,  and  interviews,  we  found  that 
while  anecdotal  evidence  exists  and  stories  are  available,  the  vast  majority  of 
YES  Teens  either  enter  the  program  with  interests  in  mind  and  retain  those 
interests,  or  enter  the  program  with  no  clear  career  path  in  mind  and  leave  with 
remaining  uncertainty.  The  YES  Program  does  expose  teens  to  careers  that  they 
had  not  previously  considered,  and  it  is  not  clear  how  many  teens  are  influenced 
by  that  exposure.  Further  studies  of  former  program  participants  would  be 
needed  to  determine  that  impact. 
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Discussion  of  Goals  and  Objectives 


The  Findings  Section  provided  evidence  to  address  the  first  evaluation 
question:  1)  How  does  participation  in  the  Community  STEM  Outreach  Program 
impact  its  participants?  This  Discussion  Section  addresses  the  remaining  two 
questions:  2)  Does/how  does  the  Community  STEM  Outreach  Program  meet  its 
goals  and  objectives?  and  3)  How  does  the  addition  of  50  or  more  teens  per  year 
affect  the  program’s  ability  to  meet  its  goals? 

HOW  DOES  THE  COMMUNITY  STEM  OUTREACH  PROGRAM  MEET 
ITS  GOALS  AND  OBJECTIVES? 

We  begin  this  section  by  matching  the  objectives  (letters)  to  the  goals 
(numbered).  Each  goal  is  then  addressed  separately. 


Project  Goals  &  Objectives 

1 .  Expand  the  YES  Program  to  reach  more  youth,  increase  diversity  of  the 
youth,  and  strengthen  STEM  content  focus 

a.  Increase  the  number  of  teens  participating  in  the  SLSC  YES  Program 
by  50 

b.  Increase  the  staff  size  beginning  to  support  the  increase  in  YES  teens, 
community  outreach,  and  other  expansion  efforts 

c.  Increase  the  number  of  community  partnering  organizations  to  include 
organizations  in  St.  Louis  County  with  outreach  to  more  diverse  youth 

2.  Expand  the  SLSC  outreach  to  schools,  particularly  high  school  science 
teachers  and  school  counselors,  to  reach  more  students  and  teachers  and 
to  develop  new  models  of  outreach 

a.  Reach  new  school  audiences  with  existing  and  new  SLSC 

programming,  focusing  on  school  counselors  and  high  school  science 
teachers 

3.  Create  a  new  model  for  partnership  with  the  US  Navy  to  include  veterans 
as  well  as  active  duty  and  reserve  personnel  in  STEM  education  outreach 
programming 

a.  Develop  new  opportunities  for  partnership  between  SLSC  staff  and 
Navy  personnel  to  support  the  YES  Program  and  other  SLSC  outreach 
activities 
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Project  Goals  &  Objectives 

4.  Create  and  formalize  a  model  for  outreach  to  and  inclusion  of  members 
of  the  science  community  in  STEM  education  programming 

a.  Develop  and  formalize  opportunities  for  involvement  of  practicing  and 
retired  scientists  in  the  community 

5.  Improve  reflective  practice  of  all  educators  at  SLSC  to  create  a  cadre  of 
leaders  for  national  outreach  to  other  science  centers 

a.  Strengthen  the  reflective  practice  of  YES  staff  and  SLSC  educators 
through  additional  training  and  ongoing  support 

b.  Train  and  support  a  cadre  of  STEM  education  leaders  who  can  train 
others  in  effective  strategies  to  build  programs  that  are  community 
relevant,  youth  development  focused,  and  strong  in  STEM  content 

c.  Create  electronic,  multimedia  documentation  of  all  Community  STEM 
Outreach  activities  and  staff  reflections  for  support  of  expansion 
efforts 

6.  Formalize  processes  and  collect  metrics  to  measure  YES  Program  short¬ 
term  and  long-term  success 

a.  Conduct  evaluation  to  support  and  provide  evidence  of  the  success  of 
the  program  and  to  identify  challenges 

7.  Codify  a  system  for  community  STEM  outreach  beyond  YES 

8.  Conduct  research  and  evaluation  to  provide  evidence  of  the  success  of 
the  program  and  to  identify  challenges 

a.  Identify  research  questions  related  to  the  Community  STEM 
Outreach,  and  create  strategies  for  moving  forward  with  that 
research,  including  seeking  additional  funding  for  such  research 

9.  Create  a  strategic  plan,  resources  and  model  for  national  expansion  of 
the  SLSC  Community  STEM  Outreach  Program  and  begin 
implementation 

a.  Create  a  strategic  plan  for  national  expansion  of  the  SLSC 
Community  STEM  Outreach  Program 

b.  Identify  resources,  including  national  partners,  for  national  outreach 
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Goal  1.  Expand  the  YES  Program  to  reach  more  youth,  increase  diversity  of 
the  youth,  and  strengthen  STEM  content  focus 

Goal  1  addresses  three  aspects  of  the  YES  Program  -  number  of  youth 
served,  diversity  of  youth,  and  content  covered.  The  matching  objectives  include 
an  increase  of  staff  to  support  these  three  aspects. 

As  seen  in  Figure  6  on  page  19,  the  cohort 
of  New  Teens  beginning  in  201 1  (after  ONR 
funding  began)  was  123,  a  full  68  more  teens 
than  joined  YES  in  2010.  Fifty  of  those  were 
covered  by  ONR  funding.  The  number  dropped 
to  66  in  2012,  and  dropped  to  below  previous 
levels  with  51  in  2013.  Thus,  the  Community 
STEM  Outreach  Program  met  its  first  objective 
of  increasing  the  number  of  participating  teens 
by  50  for  the  first  year,  but  was  unable  to 
sustain  the  increase. 

The  diversity  of  the  youth  increased  with 
the  cohort  of  201 1  and  ONR  funding. 

Immediately  prior  to  that,  program  staff’s 

records  indicate  all  YES  Teens  with  known  ethnicity  were  African  American.  A 
few  had  no  ethnicity  indicated.  The  data  in  Table  3  and  Figure  4  on  page  17 
show  evidence  of  increased  diversity.  Using  the  same  definitions  of  ethnicity, 
Table  20  compares  data  collected  on  participants  by  cohort. 


Table  20.  Percent  ethnicity  of  YES  Teens  by  cohort 


Ethnicity 

2010 

2011 

2012 

2013 

Black  or  African  American 

94.5 

73.2 

80.3 

90.2 

White/Caucasian 

0.0 

4.1 

3.0 

5.9 

Asian 

0.0 

4.9 

3.0 

0.0 

Hispanic 

0.0 

0.0 

0.0 

2.0 

Multiple  Races 

0.0 

2.4 

0.0 

2.0 

Data  not  available 

5.5 

15.4 

13.6 

0.0 

It  should  be  noted  that  these  data  are  by  cohort,  so  at  any  point  in  time  there 
may  be  more  or  less  diversity  as  teens  choose  to  participate  or  not.  The  trends 
over  time  in  Table  20  reflect  the  new  partnership  created  with  the  International 
Studies  Program  at  St.  Louis  Public  Schools  to  bring  in  recent  immigrants  of 
Asian  decent.  It  is  unclear  why  these  numbers  declined  in  2012  and  returned  to 
zero  in  2013.  Most  of  the  teens  that  entered  in  2011  and  2012  remained  active  in 
the  program  in  2013. 

To  support  the  increased  number  of  youth  in  spring  201 1  and  the 
diversification  of  the  participants,  SLSC  hired  a  project  manager  and  added  three 
full-time  and  three  part-time  educators  in  February  201 1 .  As  the  educators 
began,  they  were  sometimes  treated  differently  and  separated  from  the  larger 
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group  of  long-time  YES  educators.  In  their  first  semester,  these  new  staff 
members  were  assigned  the  New  Teen  group  to  develop  curriculum  and  facilitate 
the  Learning  Labs.  One  of  these  new  educators  left  during  spring  201 1  and 
another  left  in  summer  201 1 ;  neither  position  was  replaced.  Sixteen  part-time 
educators  were  hired  for  the  summer  to  support  YES  teens  working  with  younger 
children  from  community  organizations,  and  to  help  document  the  YES  program 
model. 

The  project  manager  position  was  added  to  oversee  all  aspects  of  the 
Community  STEM  Outreach  project,  including  the  community  outreach  and 
national  expansion.  Based  on  observations  of  meetings  and  Learning  Labs,  the 
manager  focused  on  supervising  new  staff. 

In  spring  2012,  the  institution  underwent  financial  restructuring  which  included 
the  laying  off  of  staff  throughout  the  SLSC.  Some  YES  educators  and  managers 
left  the  program  through  layoffs.  A  few  resigned  which,  according  to  interviews 
with  staff,  was  due  to  perceived  lack  of  job  security.  Table  21  provides  the 
staffing  levels  over  time  given  in  average  Full-time  Equivalent  (FTE)  by  year.  For 
the  first  year  of  the  project,  the  YES  Program  increased  number  of  staff  to 
support  the  increased  number  of  youth.  This  was  not  sustained  in  the  following 
years. 

Table  21 .  Total  FTE  over  time 


Spring 

2011 

Spring 

2012 

Spring 

2013 

Educators 

11.95 

12.50 

9.40 

Managers 

4.35 

3.23 

2.11 

Support  staff 

2.74 

0.24 

1.01 

Vice  President 

0.95 

1.00 

0.41 

Total 

19.99 

16.97 

12.93 

Overall  there  were  fewer  administrators  and  support  staff  in  the  YES  Program 
after  restructuring,  yet  all  the  management  functions  remained  (curriculum 
development,  staff  professional  development  and  support,  data  management, 
grants  management,  and  administrative  roles  within  the  larger  SLSC 
organization).  Many  administrative  responsibilities  were  distributed,  primarily 
among  the  senior  educators,  who  had  also  picked  up  more  responsibility  with 
increased  numbers  of  YES  Teens.  At  the  same  time,  administrators  picked  up 
the  responsibility  of  leading  the  New  Teen  Learning  Labs  and  some  of  the  tasks 
previously  assigned  to  support  staff. 

As  the  new  educators  were  hired  in  spring  201 1  with  funding  from  ONR,  YES 
leadership  made  a  conscious  effort  to  hire  staff  with  strong  STEM  backgrounds 
to  strengthen  the  STEM  focus  of  the  program.  These  staff  members  came  with 
undergraduate  and  sometimes  advanced  degrees  in  various  STEM  fields.  By 
having  educators  with  strong  youth  development  experience  work  with  educators 
with  solid  STEM  content  knowledge,  the  program  strengthened  its  STEM  focus. 
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To  support  the  development  of  math  skills  among  YES  Teens,  who  were 
often  tracked  into  lower  level  math  courses  instead  of  college  prep  courses, 

SLSC  leadership  suggested  YES  Teens  work  through  Khan  Academy  lessons. 
These  lessons  are  individual  math  lessons  in  which  students  can  work  at  their 
own  pace.  This  idea  was  implemented  without  a  strong  educational  plan  in  place 
to  support  it,  and  without  adequate  staff  preparation.  It  was  difficult  to  determine 
impact  of  this  aspect  of  the  program.  Some  educators  continued  to  have  YES 
Teens  in  their  component  work  through  Khan  Academy  lessons  into  2013. 

Goal  2.  Expand  the  SLSC  outreach  to  schools,  particularly  high  school 
science  teachers  and  school  counselors,  to  reach  more  students  and 
teachers  and  to  develop  new  models  of  outreach 

The  SLSC  continued  to  offer  educational  programs  to  schools  throughout  the 
ONR  grant  period.  YES  Program  educators  offered  Family  Math  programs  and 
other  STEM  long-term  and  short-term  programs  to  area  students  and  their 
families.  Educators  offered  occasional  professional  development  to  teachers  in 
some  area  school  districts.  At  the  same  time,  staff  from  the  SLSC  School 
Programs  Department  offered  short-term  programs  to  schools. 

In  early  discussions  to  plan  the  Community  STEM  Outreach  project, 
stakeholders  expressed  interest  in  bringing  the  YES  and  School  Programs 
departments  of  SLSC  together  to  reach  more  high  school  students  and  their 
teachers  and  school  counselors.  Part  of  this  outreach  was  to  provide  school  staff 
with  access  to  STEM  educational  resources,  including  opportunities  offered  by 
the  Navy.  Movement  toward  this  goal  was  sporadic,  with  occasional 
conversations  and  plans  that  fell  short  of  the  goal  of  “new  models  of  outreach.” 

Work  toward  this  goal  appeared  to  have  been  affected  by  staff  layoffs  in  both 
the  Community  Science  Outreach  and  School  Programs  departments,  as  well  as, 
institutional  reorganization  and  reassessment  during  the  project  timeframe. 

Goal  3.  Create  a  new  model  for  partnership  with  the  US  Navy  to  include 
veterans  as  well  as  active  duty  and  reserve  personnel  in  STEM  education 
outreach  programming 

Personnel  from  the  US  Navy  and  other  branches  of  the  military  spoke  with 
YES  Teens  on  several  occasions  as  guest  speakers.  While  some  volunteers  that 
worked  directly  with  the  teens  on  a  more  sustained  basis  did  have  prior  military 
experience,  there  was  no  active  program  to  recruit,  train,  or  support  active, 
reserve  or  retired  Navy  personnel  as  volunteers  in  the  YES  Program. 

In  addition  to  adult  volunteers,  Navy  involvement  included  two  youth  from  the 
Navy’s  Sea  Cadet  program  as  YES  Teens.  Additional  YES  Teens  were  students 
from  Cleveland  NJROTC  High  School,  the  St.  Louis  Public  Schools’  Naval  Junior 
ROTC  school. 
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Goal  4.  Create  and  formalize  a  model  for  outreach  to  and  inclusion  of 
members  of  the  science  community  in  STEM  education  programming 

Members  of  the  science  community  were  involved  in  the  YES  Program  in  a 
number  of  ways.  University  students  (undergraduate  and  graduate)  volunteered 
in  Learning  Labs,  particularly  in  the  Neuroscience  component.  Parents,  who  were 
current  or  retired  scientists,  volunteered  in  Learning  Labs,  such  as  three  mothers 
of  YES  Teens  who  assisted  with  the  Biofuels  and  Energy  component  during  the 
2011-2012  school  year.  Volunteers  from  local  organizations,  like  the  Webster 
Groves  Nature  Study  Society,  helped  with  YES  Teen  projects,  like  identifying 
butterflies  in  the  C3  component.  Working  professionals  from  area  businesses 
spoke  as  special  guests,  such  as  an  intellectual  property  attorney  who  spoke  with 
teens  about  types  of  income,  and  received  rave  reviews  from  teens. 

Professionals  from  AT&T  mentored  and  provided  technical  guidance  to  the 
Robotics  component.  Another  example  was  a  professional  from  Hosco  Farms 
who  began  helping  with  occasional  Learning  Labs,  then  led  a  summer 
component,  and  began  working  with  YES  leadership  on  rethinking  the  curriculum 
to  focus  on  issues  around  food,  as  he  and  groups  of  teens  designed  and  built 
aquaponic,  hydroponic,  and  aeroponic  systems  at  the  Taylor  Community  Science 
Resource  Center  (home  to  the  YES  Program). 

In  August  2012,  a  plan  was  in  place  to  establish  a  formal  program  for 
recruiting  scientists.  When  the  Senior  Educator  in  charge  of  that  plan  left  the 
SLSC  for  another  job,  the  plan  was  not  implemented.  Instead,  during  the 
evaluation  period,  educators  of  STEM  components  continued  their  practice  of 
inviting  scientists  they  knew  through  established  networks  and  structures. 

Thus,  while  scientists  were  involved  in  the  YES  Program,  no  new  formalized 
model  was  implemented  to  recruit,  train,  or  support  volunteers  from  the  science 
community. 

Goal  5.  Improve  reflective  practice  of  all  educators  at  SLSC  to  create  a 
cadre  of  leaders  for  national  outreach  to  other  science  centers 

The  three  objectives  associated  with  Goal  5  cover  different  aspects  of 
reflective  practice,  leadership,  and  expansion. 

Objective  A:  Strengthen  the  reflective  practice  of  YES  staff  and  SLSC 
educators  through  additional  training  and  ongoing  support.  Professional 
development  (PD)  of  YES  educators  took  several  forms.  Experts  were  brought  in 
to  provide  training  in  some  cases,  such  as  Science  Museum  of  Minnesota 
President  Eric  Jolly  covering  the  community  standard,  of  which  summer  interns 
spoke  very  highly.  Additionally,  staff  from  the  Program  in  Education,  Afterschool 
and  Resiliency  (PEAR)  provided  training  on  the  Dimensions  of  Success  (DoS) 
program  observation  tool.  At  times,  educators  were  sent  to  workshops  and  PD 
elsewhere,  such  as  staff  going  to  NPASS  (National  Partnerships  for  Afterschool 
Science)  training.  A  third  form  of  PD  occurred  when  educators  led  PD  for  each 
other,  such  as  workshops  led  by  the  educator  who  attended  the  NPASS  training 
and  a  book  discussion  group  on  John  Dewey’s  work  led  by  another  educator. 
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Objective  B:  Train  and  support  a 
cadre  of  STEM  education  leaders 
who  can  train  others  in  effective 
strategies  to  build  programs  that  are 
community  relevant,  youth 
development  focused,  and  strong  in 
STEM  content.  The  reflective  practice 
and  PD  described  in  relation  to 
Objective  A  focused  on  inquiry  and 
project  based  learning.  As  described 
in  formative  evaluation  reports  for  the 
Community  STEM  Outreach  project, 
training  people  to  lead  STEM 
activities  is  very  different  than  training 
the  trainers.  For  example,  educators  received  PD  to  support  them  in  facilitating 
inquiry-based  STEM  lessons  with  YES  Teens.  However,  PD  did  not  cover  how  to 
train  YES  Teens  to  lead  activities  for  children. 


Through  modeling  by  Diane  Miller  and  Colin  Wilson,  YES  Educator  and  a 
Master  Trainer  (using  NPASS)  through  the  Missouri  Afterschool  Network,  YES 
Educators  did  learn  to  train  Community  Partners  to  lead  STEM  activities  with 
their  own  youth  in  their  community  programs. 

Objective  B  was  originally  intended  to  develop  YES  Educators  as  leaders  to 
train  and  support  educators  from  other  science  centers  and  museums,  so  that 
other  educators  could  build  additional  youth  STEM  education  programs  that  were 
relevant  to  their  own  local  situations  and  youth.  This  part  of  the  objective  was  not 
implemented  during  the  evaluation  period. 

Objective  C:  Create  electronic,  multimedia  documentation  of  all  Community 
STEM  Outreach  activities  and  staff  reflections  for  support  of  expansion  efforts. 
This  objective  builds  on  the  previous  objective  and  the  intent  to  expand  the 
Community  STEM  Outreach  project’s  reach  to  other  science  center  educators.  In 
the  summer  of  201 1 ,  a  large  team  of  part-time  staff  was  hired  as  “documenters” 
to  record  photos  and  videos  of  the  program,  and  to  make  observations.  While 
large  numbers  of  photos  and  hours  of  video  were  recorded,  the  pulling  together 
of  the  data  into  a  cohesive  whole  became  a  challenge  for  YES  leaders.  In 
December  201 1 ,  the  leadership  contracted  with  Klein  Consulting  to  create  the 
documentation,  now  called  the  Circles  of  Support  web-based  multimedia  tool 
(described  on  page  15  of  this  report). 


Goal  6.  Formalize  processes  and  collect  metrics  to  measure  YES  Program 
short-term  and  long-term  success 

Activities  to  accomplish  this  goal  are  covered  throughout  this  summative 
report.  The  intent  of  this  goal  was  to  establish  processes  and  to  collect  baseline 
data  in  preparation  for  national  expansion  efforts.  The  pilot  of  the  alumni  online 
survey  was  one  example.  In  this  case,  the  Klein  Consulting  evaluation  team 
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piloted  the  survey  and  made  recommendations  for  future  alumni  surveys  and 
events. 

Challenges  arose  when  different  staff  members  over  time  were  assigned  to 
collect  and  maintain  program  data,  such  as  teen  demographic  data.  Often  these 
tasks  were  assigned  to  educators  who,  according  to  interviews  and  focus  groups, 
placed  curriculum  planning  and  working  with  the  teens  as  a  higher  priority  than 
data  collection  and  maintenance,  suggesting  the  need  for  support  staff  to  fill  this 
role.  The  challenges  faced  in  the  calculations  provided  in  this  summative 
evaluation  document  highlight  the  need  for  the  YES  Program  staff  to  establish 
and  follow  a  more  rigorous  system  of  data  collection  if  high  school  graduation 
rates  and  similar  data  will  be  needed  in  the  future.  A  follow-up  strategy  to 
maintain  current  addresses  and  email  addresses  for  alumni  is  needed.  For 
example,  some  projects  send  birthday  cards  though  the  mail  each  year  and 
those  that  are  returned  are  targeted  for  email  contact  and  address  update. 
Recommended  steps  for  future  data  collection  and  maintenance  include: 

•  Maintenance  of  current  high  school  data  for  all  YES  Teens  (some  data 
were  missing  from  the  current  database  and  some  actual  schools  differed 
from  current  records) 

•  Frequent  contact  with  all  YES  seniors  throughout  their  senior  year  to  be 
sure  they  are  on  track  for  graduation  (and  their  plans  are  known  to  staff) 

•  Maintenance  of  contact  information  and  records  for  YES  Teens  who  drop 
out  of  YES  before  or  during  their  senior  year  (to  allow  for  further 
comparison) 

Keeping  accurate  records  and  high  school  graduation  data  for  all  YES  Teens 
(including  those  who  leave  the  program)  would  allow  staff  to  better  and  more 
accurately  address  questions  from  funding  agencies  and  donors  regarding 
program  success. 

Goal  7.  Codify  a  system  for  community  STEM  outreach  beyond  YES 

To  codify  generally  refers  to  creating  or  arranging  a  system,  in  this  case  a 
system  to  take  the  YES  Program  model  to  new  youth  programs  or  youth 
programs  that  want  to  learn  from  the  YES  Program  model.  The  Circles  of 
Support  multimedia  tool  design  was  created  by  Carey  Tisdal  of  Tisdal  Consulting 
in  collaboration  with  Klein  Consulting  to  support  this  goal.  While  it  was  beyond 
the  scope  of  work  by  Tisdal  and  Klein  to  codify  a  system,  a  plan  was 
recommended  and  provided  to  YES  leaders. 

Goal  8.  Conduct  research  and  evaluation  to  provide  evidence  of  the 
success  of  the  program  and  to  identify  challenges 

Again,  evaluation  activities  to  accomplish  this  goal  are  covered  throughout 
this  summative  report.  In  addition  to  the  evaluation,  the  specific,  related  objective 
states:  Identify  research  questions  related  to  the  Community  STEM  Outreach 
Program,  and  create  strategies  for  moving  forward  with  that  research,  including 
seeking  additional  funding  for  such  research.  Klein  Consulting  provided  the 
SLSC  Leadership  with  a  separate  document  outlining  research  questions  that 
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emerged  from  the  evaluation,  and  recommendations  for  moving  that  research 
forward.  During  the  project,  SLSC  submitted  two  grant  proposals  to  the  National 
Science  Foundation  for  related  research,  though  neither  was  funded. 

Goal  9.  Create  a  strategic  plan,  resources,  and  model  for  national 
expansion  of  the  SLSC  Community  STEM  Outreach  Program  and  begin 
implementation 

The  first  step  in  this  plan  was  to  bring  together  representatives  from  science 
centers  and  museums  across  the  country.  After  the  April  2013  meeting,  the  lack 
of  follow-up  discussion  from  conference  attendees  and  guidance  from  ONR 
regarding  limited  funding  opportunities  led  the  SLSC  administration  to  conclude 
that  an  appropriate  role  for  the  institution  would  be  to  create  the  design  for  a  tool 
for  national  dissemination  (the  Circles  of  Support  website).  A  proposed 
dissemination  plan,  using  the  Circles  of  Support  tool  as  a  foundation,  was 
developed  as  part  of  the  contract  with  Klein  Consulting.  This  plan  was  provided 
to  SLSC  Leaders  with  the  summative  evaluation  report  and  the  research 
document  described  above. 

Summary  of  Goals  and  Objectives  Met 

Of  the  nine  goals  and  14  objectives,  results  indicate  that  the  YES  Program: 

1 .  Increased  the  number  and  diversity  of  YES  Teens  (Goal  1 ,  Objectives  1 
and  2),  although  increases  were  not  sustained  in  following  years. 

2.  Increased  the  number  of  Community  Partners  (Goal  1 ,  Objective  3) 

3.  Strengthened  the  STEM  Content  (Goal  1) 

4.  Strengthen  the  reflective  practice  of  YES  staff  (Goal  5,  Objective  8) 

5.  Supported  the  design  for  an  electronic,  multimedia  documentation  of  the 
YES  Program,  the  Circles  of  Support  web-based  multimedia  tool  (Goal  5, 
Objective  10) 

6.  Contracted  and  supported  the  evaluation  to  identify  challenges  and 
provide  evidence  of  success  of  the  project  (Goal  6,  Objective  1 1 ) 

7.  Identified  research  questions  related  to  the  project,  sought  additional 
funding  (though  unsuccessfully),  and  received  recommended  strategies  to 
move  forward  with  that  research  (Goal  8,  Objective  12) 

8.  Identified  a  plan  for  national  expansion  beyond  YES  (Goal  9) 

HOW  DOES  THE  ADDITION  OF  50  OR  MORE  TEENS  PER  YEAR 
AFFECT  THE  PROGRAM’S  ABILITY  TO  MEET  ITS  GOALS? 

The  addition  of  teens  revealed  issues  of  scale  related  to  the  need  for  more 
formal  structures  in  professional  development,  curriculum  development,  data 
management,  youth  participation  support,  and  a  changed  role  for  managers. 
These  issues  were  identified  in  the  Second  Annual  Evaluation  Report.  The 
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evaluation  team,  Klein  and  Tisdal,  met  with  YES  staff  in  December  2012  and 
January  2013  to  discuss  the  issues. 


Increased  numbers  of  teens  required  an  increased  number  of  full-time  and 
part-time  staff  members.  Observations  of  Learning  Labs  indicated  that  new  and 
summer  staff  were  not  always  clear  about  some  of  the  program  learning 
strategies  including  the  Learn  to  Teach  -  Teach  to  Learn  strategy  and  the 
relationship  between  teaching  topics  and  inquiry  projects  developed  by  the  teens. 
For  example,  some  of  the  part-time  summer  staff  in  201 1 ,  hired  to  document  the 
program,  struggled  to  understand  and  document  the  learning  strategies.  Some  of 
the  newer  summer  staff  in  2012  struggled  to  understand  the  philosophy  of  a 
work-based  program  and  the  importance  of  meeting  the  teens’  development 
needs.  Observations,  and  results  of  staff  focus  groups,  indicated  a  lack  of 
understanding  of  some  YES  Program  learning  strategies  by  both  managers  and 
part-time  staff  members. 

These  issues  appeared  to  be  related  to  scale;  with  a  smaller  staff,  program 
design  and  strategies  could  be  shared  through  informal  communications.  Larger 
numbers  of  teens,  requiring  larger  number  of  staff  members  and  increased  time 
by  staff  to  focus  on  those  teens’  needs,  required  the  adoption  of  more  formal 
Professional  Development.  The  addition  of  a  group  of  new  educators  at  one  time 
with  the  project  start,  rather  than  one  or  two  new  educators  replacing  educators 
who  left  the  program  through  retirement  and  resignation,  created  an  even 
stronger  need  for  formalized  Professional  Development.  The  Circles  of  Support 
program  documentation  element  and  website  could  fill  this  need  in  the  future. 

Increased  numbers  of  teens  required  changes  to  curriculum  for  the  New  Teen 
Learning  Labs,  College  Prep,  and  STEM  components  to  accommodate  the  larger 
numbers  of  participants.  This  revealed  the  need  for  changes  in  curriculum 
development,  specifically  an  overall  curriculum  to  align  with  YES  goals  and 
program  framework,  support  for  educators  in  the  development  of  component  and 
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College  Prep  curriculum,  and  someone  to  ensure  quality  and  consistency  of 
implementation.  While  such  roles  would  normally  be  assigned  to  a  single 
manager,  after  staff  turnover  this  was  assigned  to  various  managers  and  Senior 
Educators  resulting  in  inconsistent  implementation,  curriculum  quality,  and  staff 
support. 

As  numbers  of  teens  increased,  so  did  the  need  to  track  data  such  as 
attendance  records,  payroll  records,  demographic  data,  and  contact  information. 
When  the  Community  STEM  Outreach  Program  began,  one  manager,  one 
Senior  Educator,  and  one  support  staff  member  filled  these  roles;  however,  with 
staff  reductions  these  roles  were  filled  by  a  variety  of  staff  with  resulting  gaps  in 
data. 

In  the  earlier  years  of  the  YES  Program,  the  educators  called  teens  within  a 
few  days  if  they  did  not  report  for  work  or  call  to  explain  their  absence.  With  the 
increase  in  number  of  teens  and  since  some  educators  did  not  understand  the 
importance  of  this  strategy,  educators  dropped  this  practice,  perhaps  due  to  the 
increase  in  other  priorities.  As  a  result,  follow-up  with  teens  that  missed  Learning 
Labs  or  dropped  out  of  the  program  was  inconsistent. 

Based  on  observations  of  staff  meetings  and  interviews  with  managers  and 
educators,  it  appeared  that  the  addition  of  50  or  more  teens  also  required 
managers  to  spend  more  time  on  program  logistics;  that  is,  handling  the  Human 
Resource  interface,  scheduling  staff  and  teens,  and  reporting  staff  and  teen 
hours  for  payment.  Unfortunately,  the  number  of  program  managers  was  cut  at 
the  point  when  additional  management  time  was  needed  to  support  larger 
numbers  of  teens  by  providing  increased  staff  professional  development, 
curriculum  development,  and  data  management.  While  managers  still  dropped  in 
and  out  of  classrooms  after  the  staff  reductions,  managers’  focus  shifted  toward 
management  and  logistical  issues  rather  than  observing  educator  and  teen 
interaction  and  providing  feedback  and  support  for  teaching  and  learning  issues. 

In  summary,  the  addition  of  teens  highlighted  the  need  for  more  formalized 
management  structures  and  practices,  increased  manager  time  in  scheduling 
and  logistics,  formal  professional  develop  for  consistent  implementation, 
additional  curriculum  development,  consistent  data  management,  and  renewed 
support  for  youth  participation. 


Summary  and  Conclusions 

The  evaluation  of  the  Community  STEM  Outreach  Program  was  designed  to 
address  three  questions: 

1 .  How  does  participation  in  the  Community  STEM  Outreach  Program 
impact  its  participants? 

2.  Does/how  does  the  Community  STEM  Outreach  Program  meet  its 
goals  and  objectives? 

3.  How  does  the  addition  of  50  or  more  teens  per  year  affect  the 
program’s  ability  to  meet  its  goals? 
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The  Short-term  Impacts  of  the  program  included  high  school  graduation  rates 
and  post-secondary  plans.  We  found  that  YES  Teens  who  remain  in  the  program 
until  their  senior  year  in  high  school  graduate  at  a  higher  rate  than  their  peers  in 
the  same  area  schools,  with  96.36%  of  YES  Teens  that  could  be  contacted 
graduating  in  2012  and  100%  in  2013.  From  the  graduating  class  of  2012,  55% 
of  YES  Teens  planned  to  attend  a  four-year  college  or  university,  and  in  2013  the 
percent  was  46.  Two  issues  were  identified  in  arriving  at  high  school  graduation 
and  post-secondary  plans  data.  The  YES  Program  does  not  track  teens  that  drop 
out  of  the  program,  and  staff  members  do  not  maintain  accurate  contact 
information  to  contact  teens  after  graduation.  In  2012,  12%  of  high  school  seniors 
were  not  reached  and  in  2013  the  number  grew  to  23%,  perhaps  accounting  for 
the  perceived  decrease  in  percent  of  teens  attending  a  four-year  institution  after 
high  school  graduation. 

The  evaluation  team  explored  the  impact  of  the  program  on  attitudes  toward 
science  and  scientists  using  a  modified  version  of  the  Test  of  Science-Related 
Attitudes  (TOSRA2).  Results  indicated  that  after  16  months  of  participation  in 
YES,  teens  were  more  likely  to  1 )  recognize  the  benefits  and  drawbacks  of 
scientific  advances  to  society,  2)  see  scientists  as  real  people  rather  than  media- 
produced  stereotypes,  3)  indicate  enjoyment  of  their  lessons  in  school  science 
classes,  and  4)  be  interested  in  science  out  of  school  and  outside  of  the  YES 
Program,  and  were  slightly  more  likely  5)  to  have  adopted  the  attitudes  of 
scientists.  After  16  months,  YES  Teens  were  less  likely  to  view  experimentation 
and  inquiry  as  a  way  to  gain  understanding  of  the  natural  world,  and  were  slightly 
less  interested  in  pursuing  a  science  related  career. 

The  Long-Term  Impacts  of  the  YES  Program  were  more  difficult  to  assess. 

An  online  survey  of  former  YES  Teens  received  only  22  valid  responses  by  the 
deadline.  Results  from  the  small  sample  showed  great  promise  for  future  studies. 
Twenty  of  the  22  respondents  graduated  from  high  school  and  two  earned  a 
GED.  After  high  school,  18  of  the  respondents  (82%)  attended  a  trade  school, 
college  or  university.  Five  of  those  had  graduated  from  college,  with  two  in 
graduate  school  at  the  time  of  the  pilot  survey.  Eleven  were  still  in  trade  school 
(1  participant)  and  college  (10  participants). 

When  asked  what  influenced  their  choice  of  career,  family  and  school 
experiences  were  most  prominent  with  YES  experiences  a  close  third. 
Respondents  reported  that  the  YES  Program  was  “helpful”  (an  average  of  3.80 
on  a  5-point  scale)  in  preparing  them  for  post-secondary  education.  When  asked 
how  well  the  program  prepared  them  for  the  workforce,  the  resulting  average  was 
4.53  on  the  same  5-point  scale,  indicating  that  the  YES  Program  was  “very 
helpful”  in  workforce  preparation.  When  asked  how  influential  the  YES  Program 
was  on  college  and  career  choices,  the  average  was  3.88,  “helpful”.  Results 
indicate  that  networking,  teambuilding,  and  teaching  younger  children  were  the 
aspects  of  the  program  that  influenced  respondents  the  most.  When  asked  about 
the  influence  of  the  YES  Program  on  their  attitudes,  results  showed  that  the 
greatest  influences  were  on  attitudes  toward  adults,  learning,  young  children, 
workplace  policies  and  procedures,  and  teachers. 
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The  program  met  many  of  the  project’s  goals  and  objectives.  Personnel  and 
budget  changes  at  SLSC  created  challenges  for  meeting  others.  Some  goals 
were  met  prior  to  those  changes,  but  were  not  sustained. 

Goal  1:  The  YES  Program  expanded  to  reach  more  youth,  increase  diversity 
of  the  youth,  and  strengthen  STEM  content  focus,  although  the  incoming  number 
of  youth  and  the  diversity  of  youth  began  to  return  to  pre-grant  levels  by  the  end 
of  the  grant  period. 

Goal  2:  Expansion  of  outreach  to  schools  focused  on  diversifying  the  YES 
Teens.  Outreach  to  high  school  science  teachers  and  school  counselors  to 
provide  resources  received  minimal  focus,  perhaps  due  to  the  institutional 
changes. 

Goal  3:  Navy  personnel  were  involved  in  the  program,  although  a  new  model 
for  partnership  to  include  veterans  as  well  as  active  duty  and  reserve  personnel 
in  STEM  education  outreach  programming  did  not  materialize  to  the  extent  that  it 
could  be  replicated. 

Goal  4:  Members  of  the  science  community  were  involved  in  the  YES 
Program  in  many  ways,  although  a  new,  formalized  model  for  outreach  to  and 
inclusion  of  members  of  the  science  community  did  not  materialize  to  the  extent 
that  it  could  be  replicated.  Individual  staff  members  within  YES  were  very 
successful  at  involving  scientists  in  STEM  components,  and  YES  staff  engaged 
scientists  in  many  activities  involving  the  YES  Teens. 

Goal  5:  The  YES  Program  strengthened  the  reflective  practice  of  YES  staff 
and  SLSC  educators  through  additional  training  and  ongoing  support.  The 
second  objective  to  “train  and  support  a  cadre  of  STEM  education  leaders  who 
can  train  others  in  effective  strategies  to  build  programs  that  are  community 
relevant,  youth  development  focused,  and  strong  in  STEM  content”  was  originally 
intended  to  move  YES  Educators  into  a  position  to  train  and  support  educators 
from  other  science  centers  and  museums,  so  that  those  educators  could  build 
additional  youth  STEM  education  programs  that  were  relevant  to  their  own 
situations  and  youth.  With  the  restructuring  of  the  SLSC,  this  part  of  the  objective 
did  not  materialize.  Klein  Consulting  prepared  the  Circles  of  Support  multimedia 
tool  to  achieve  the  third  objective,  “Create  electronic,  multimedia  documentation 
of  all  Community  STEM  Outreach  activities  and  staff  reflections  for  support  of 
expansion  efforts.” 

Goal  6:  While  some  processes  to  measure  YES  Program  short-term  and 
long-term  success  were  put  into  place,  a  formalized  process  for  ongoing  data 
collection  and  management  was  not  accomplished. 

Goal  7:  The  goal  of  codifying  a  system  for  community  STEM  outreach  beyond 
YES  was  established  to  take  the  YES  Program  model  to  other  youth  programs. 
Toward  that  end,  Carey  Tisdal,  with  support  from  Christine  Klein,  designed  the 
Circles  of  Support  multimedia  tool  and  recommended  a  plan  for  outreach. 
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Goal  8:  The  summative  evaluation  provides  evidence  of  the  success  of  the 
program  and  identifies  challenges.  Klein  Consulting  provided  recommendations 
for  future  research. 

Goal  9:  The  SLSC  convened  a  meeting  of  representatives  from  nine  science 
centers  and  museums  across  the  country,  though  no  strategic  plan  for  national 
expansion  emerged.  Through  a  contract  with  SLSC,  Klein  Consulting  proposed  a 
dissemination  plan  with  the  Circles  of  Support  tool  as  a  foundation. 

The  addition  of  more  than  50  YES  Teens  presented  challenges  that  were 
further  impacted  by  institutional  changes.  Roles  of  managers  changed,  and  there 
were  increased  needs  for  formal  structures  in  professional  developed,  curriculum 
development,  data  management,  and  youth  participation  support. 

Conclusion 

The  Deliberate  Design  of  the  YES  Program  lays  a  strong  foundation  for  youth 
development.  Many  positive  impacts  were  found:  higher  than  average  high 
school  graduate  rates,  large  numbers  of  teens  planning  to  continue  their 
education  beyond  high  school  graduation,  and  alumni/alumnae  who  stated  that 
the  YES  Program  was  helpful  in  preparing  them  for  post-secondary  education 
and  very  helpful  in  preparing  them  for  the  workforce.  Participants  improved  their 
attitudes  toward  science  and  scientists. 

At  the  same  time,  the  program  was  challenged  by  personnel  turnover, 
sometimes  resulting  in  inconsistent  application  of  the  Deliberate  Design  of  the 
program.  Thus,  we  learned  through  the  evaluation  that  consistent  systems  of 
professional  development  and  sufficient  managerial  staffing  are  needed  to 
maintain  the  Deliberate  Design.  In  addition,  we  learned  the  vulnerability  of 
program  functions  with  the  loss  of  institutional  memory  due  to  staff  turnover. 

The  YES  Program,  though  challenged  at  times  by  a  number  of  factors 
described  in  the  summative  evaluation,  offers  a  model  for  other  youth  STEM 
programs.  As  with  all  programs,  ongoing,  sustainable  funding  for  the  program 
requires  telling  the  YES  story  to  stakeholders.  Through  the  evaluation,  we  found 
that  the  development  and  maintenance  of  program  records,  which  is  key  to  this 
sustainability,  needs  to  be  a  focus  as  the  program  moves  forward. 

This  summative  report  provides  part  of  the  YES  story  at  a  moment  of  change 
and  challenge.  As  the  program  continues  to  grow  and  change,  this  evaluation 
team  recommends  that  the  Deliberate  Design  of  the  program  remains  a  solid 
foundation.  This  means  that  clear  rationale,  based  on  research  and  best 
practices,  needs  to  be  developed  for  any  changes  with  an  eye  toward  how  these 
changes  may  affect  impacts  documented  in  this  evaluation. 


Feedback  on  this  report  and  questions  about  the  evaluation  can  be  sent  to 
Christine  (Kit)  Klein,  evaluation  consultant,  at  ckleinconsutling@gmail.com. 
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Appendix  A  -  Vignettes  from  Second  Annual 
Evaluation  Report 


The  vignettes  here  are  taken  from  the  Community  STEM  Outreach  Second 
Annual  Evaluation  Report.  These  are  included  in  this  summative  report  to  provide 
a  more  complete  picture  of  the  program. 

What  happens  in  a  typical  component  during  the  school  year? 

The  following  story  comes  from  one  morning  component  group  on  a  cool, 
sunny  Saturday  in  October  201 1 .  George1,  with  a  social  work  background,  and 
Doug,  with  a  background  in  engineering,  lead  the  teens  through  a  series  of 
activities. 


School  Year  Component  Experience  Vignette 

The  YES  teens  slowly  arrive  in  their  red  YES  t-shirts  and  gather  in  the  lobby 
of  the  TCSRC.  Most  struggle  to  look  professional,  though  others  prefer  to  dress 
and  act  more  casual.  Many  take  advantage  of  the  cereal,  milk,  and  fruit  the  staff 
laid  out  for  them  in  the  kitchen.  At  8:30,  nine  members  of  the  astronomy  group 
sign  an  attendance  sheet  as  they  enter  their  classroom.  They  grab  their  journals 
from  a  milk  crate  and  take  a  seat  at  a  table.  George  and  Doug  greet  them 
individually  as  they  enter.  George  reviews  the  Word  of  the  Day  -  apogee,  and 
then  discusses  the  Fact  of  the  Day  regarding  Newton  and  the  reflector  telescope. 
A  brainteaser  follows  with  “H,  I,  J,  K,  L,  M,  N,  O”  written  on  the  board. 

Once  everyone  catches  on,  “H  to  O,  oh  I  get  it!”,  the  group  covers  a  few 
announcements  and  begins  writing  in  their  journals  as  George  turns  on  the  CD 
player.  “For  your  journaling  pleasure,  we  have  a  continuation  of  the  Soulard 
Blues  Band.  ” 

After  they’ve  written  about  experiences  in  their  lives,  their  personal 
reflections,  for  about  10  minutes,  George  leads  them  through  a  review  of 
previous  activities  on  the  angle  of  the  sun.  Today  they  will  view  sunspots  he  tells 
them.  Once  George  has  shown  them  the  homemade  tool  they  will  use,  the  teens 
visit  the  supply  table  where  Doug  and  George  have  laid  out  cardboard,  foil,  and 
tape.  They  work  easily  in  pairs,  chatting  and  teasing  each  other  in  a  manner  that 
demonstrates  comfort. 

By  9:00,  with  tools  in  hand,  the  group  heads  to  the  parking  lot.  The  wind  and 
chill  in  the  air  catch  some  offguard,  and  they  don’t  hesitate  to  point  it  out.  They 
quickly  draw 


1  Pseudonyms  are  used  for  all  characters  even  though  most  of  the  YES  staff  will  recognize 
themselves  and  each  other.  Since  other  members  of  the  learning  community  may  read  this 
report,  and  since  we  promised  the  Institutional  Review  Board  (IRB),  anonymity  is  maintained. 
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the  sun’s  reflection  on  white  paper  and  clipboards  with  the  guidance  of  George 
and  Doug,  and  then  return  to  their  classroom. 

George  explains  that  they  are  prototyping 
activities  they  will  use  later  with  Science 
Center  visitors,  and  they  will  collect  data 
over  time. 

They  shift  quickly  to  another  activity. 

Doug  begins  with  a  discussion  about  rockets  - 
what  are  they,  how  do  they  work,  what  is 
inertia?  He  explains  to  the  teens  that  this 
discussion  is  an  assessment  to  learn 
what  the  teens  know.  He  follows  the 
discussion  with  a  short  article  teens  are  asked  to  read  from  the  netbook 
computers  on  each  table.  In  their  discussion,  it  is  clear  that  most  read  the  article. 
Equally  clear  are  a  few  misconceptions  held  by  the  teens.  One  teen  asks  if 
people  really  went  to  the  moon,  and  another  says,  “I  think  we  need  to  do  it  again 
[go  to  the  moon]  so  I  can  go  and  then  I’d  know  [if  they  really  went].  ” 

Soon  Doug  reintroduces  the  Word  of  the  Day  as  he  and  George  tell  the  teens 
they  will  design,  build,  and  launch  their  own  rockets,  and  will  measure  the 
apogee.  By  9:30,  pairs  of  teens  are  creating  rockets  from  colored  paper,  PVC 
tubes,  and  masking  tape.  George  demonstrates  how  to  make  a  nose  out  of  the 
paper,  but  the  teens  design  the  rest  of  their  rockets  on  their  own.  As  they  work, 
George  and  Doug  move  around  the  room  asking  questions  about  the  science 
involved  (“What  is  the  point  of  sealing  the  top  of  the  rocket?”)  and  inquiring  about 
their  designs  (“Why  did  you  decide  to  use  four  fins?”). 

Once  the  rockets  are  ready  to  test,  the  teens 
walk  across  a  busy  intersection  to  Science 
Corner,  a  large  lot  owned  by  the  science  center 
and  used  by  YES.  In  pairs,  one  teen  holds 
their  rocket  while  the  other  stomps  on  a  two- 
liter  plastic  bottle  to  send  their  rocket  soaring. 
George  walks  among  the  pairs  asking  about 
design  features  and  suggesting  they  test  other 
ideas.  As  teens  compete  to  see  which  rocket 
will  go  highest  and  furthest,  they  try  different 
ideas  such  as  the  angle  of  the  launch. 

Upon  return  to  their  classroom,  George  asks 
what  design  features  worked  and  what  didn’t 
work,  making  lists  on  the  board.  After  discussing  many  ideas,  George  tells  the 
teens  that  this  activity  will  engage  multiple  ages,  thus  bringing  them  back  to  the 
idea  of  testing  these  activities  for  use  with  visitors  later. 

Keeping  the  teens  moving,  Doug  and  George  have  the  teens  shift  gears  to 
focus  on  review  of  science  articles  with  half  of  the  group  reading  an  article  on  a 
pee-recycling  system  used  by  NASA  and  half  reading  another  regarding  iPhones 
and  the  space  shuttle.  Each  group  reads  and  discusses  their  article,  and  then 
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summarizes  the  article  for  the  rest  of  the  teens.  George  challenges  their  thinking 
and  asks  questions  to  generate  conversation  around  details  in  the  articles. 

At  the  request  of  one  science  center  executive,  components  use  the  Khan 
Academy  online  to  support  teens  in  developing  math  skills.  George  has  the  teens 
individually  log  into  the  site  on  the  netbooks  provided.  The  attention  of  most  of 
the  teens  appears  to  wander  as  they  look  around  the  room  and  at  cell  phones 
and  occasionally  talk  with  each  other.  They  appear  bored  with  this  “school-like” 
activity,  though  George  and  Doug  offer  individual  encouragement.  Soon  George 
has  the  group  exit  Khan  Academy  and  begin  to  blog  about  the  rocket  activity  and 
the  articles  they  read. 

After  a  1 5-minute  break,  the  teens  go  to  their  college  prep  groups  and  sign  in. 
The  freshman/sophomore  groups  are  meeting  together  in  the  Jolly  meeting  room 
to  play  College  Jeopardy.  Projected  on  the  screen  are  the  categories  Colleges, 
College  Sports,  $  for  College,  Fun  Trivia,  and  Where  Am  I?  A  surprising  number 
of  teams  know  (or  guess)  the  oldest  college  in  the  US,  though  one  team 
suggests  it  is  Harris-Stowe.  Five  teams  know  that  Mizzou  (the  University  of 
Missouri)  “invented”  homecoming. 

Once  the  game  ends,  the  morning  group  of  teens  leaves  as  the  afternoon 
group  goes  to  their  component  groups.  It’s  been  a  busy  day  for  the  teens,  and  a 
long  one  for  the  staff. 


What  happens  during  a  typical  day  in  the  summer? 

YES  teens  worked  in  several  different  locations  during  the  summer  of  2012, 
yielding  several  different  stories.  Three  stories  are  told  here:  one  for  the  new 
teens  working  with  community  groups  at  the  Taylor  Community  Science 
Resource  Center  (TCSRC)  (the  Summertime  Science  component),  one  for  the 
teens  in  the  main  SLSC  building  working  with  community  groups  and  visitors  (the 
Astronomy  component),  and  a  third  story  for  the  teens  developing  exhibit 
prototypes  first  at  Compton-Drew  and  then  in  the  exhibit  galleries  (Exhibit  Lab 
component). 

Summertime  Science 

Summertime  Science  Vignette 

By  8:45  on  this  June  Wednesday,  teens  fill  the  lobby  of  the  TCSRC  as  they 
wait  to  clock  in  at  an  electronic  time  clock.  By  9:00,  everyone  is  moving  to  their 
classroom  to  sign  in  and  put  on  their  official  black  YES  aprons.  Today  is  a  big 
day,  the  first  day  that  children  from  community  organizations  will  participate  in 
activities  led  by  the  YES  teens. 

In  one  classroom,  Cheryl  has  12  teens  set  up  their  activities.  Each  of  three 
groups  of  teens  has  prepared  an  activity  to  share  with  the  younger  children,  and 
the  teens  run  through  their  plans  before  the  children  arrive.  They  begin  their 
“rehearsal”  with  an  introduction  by  a  teen  that  hasn’t  had  the  opportunity  to  give 
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one  yet.  They  discuss  how  they  will  separate  the  children  into  three  groups  to 
rotate  through  three  different  activities. 

When  Cheryl  has  to  leave  the  room,  the  teens  continue  without  her,  preparing 
for  the  activities.  The  leaders  for  a  bingo  activity  focusing  on  plants  have  the 
other  teens  line  up  and  move  into  the  hall  just  as  they  plan  to  do  with  the 
children.  Back  in  the  classroom,  they  quickly  review  the  balloon  and  journal- 
creation  activities  as  it  approaches  10:00. 

Soon  the  building  is  filled  with  children,  including  middle  school  students  from 
nearby  Compton-Drew  and  two  day  camps  with  elementary  school  aged  children. 
Twenty-one  of  the  middle  school  students  join  Cheryl’s  teens  who  divide  them 
into  three  groups  after  explaining  the  rotations. 

One  group  of  seven  students  gathers  around 
three  tables  to  create  balloon  terrariums  by  filling 
balloons  with  soil,  water,  and  seeds.  The  students 
struggle  with  blowing  up  the  balloons  and  tying 
them  off,  but  the  YES  teens  offer  help  as  needed. 

At  another  group  of  tables,  seven  more  students 
work  with  YES  teens  to  create  a  booklet  about 
plants.  In  the  hallway,  the  remaining  seven, 
students  play  a  game  of  plant  bingo  using  terms 
from  botany. 

After  moving  this  group  of  middle  school 
students  through  three  rotations,  the  community 
groups  move  to  a  new  room  and  Cheryl’s  teens  repeat  their  three  activities  with 
another  community  group.  Once  the  children  leave,  the  YES  teens  take  a  much 
needed  lunch  break. 

After  lunch,  Cheryl  facilitates  a  debriefing  of  the  morning’s  activities.  Much 
like  George  did  during  the  school  year  story  above,  Cheryl  leads  the  teens  in  a 
discussion  of  what  worked  and  what  didn’t  work  before  they  begin  to  prepare  for 
tomorrow’s  groups  of  children.  Since  this  was  their  first  day  working  with 
children,  the  teens  are  full  of  ideas  for  improvements  to  the  activities  and  their 
introduction. 

After  they  wrap  up  their  discussion  and  take  a  short  break,  the  teens  write  in 
their  journals,  saving  their  “Word  of  the  Day”  and  “Quote  of  the  Day”  for  tomorrow 
when  they  plan  to  have  a  little  more  time.  Cheryl  offers  a  writing  prompt  for  their 
journaling,  “Where  would  you  like  to  go  if  you  could  go  anywhere  in  the  world  and 
why 

After  putting  their  journals  away,  the  teens  clean  the  room  and  set  it  up  for  the 
next  day,  when  they  will  repeat  their  three  activities  with  two  new  groups  of 
children. 
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Astronomy 


Astronomy  Summer  Component  Vignette 

Inside  the  new  Boeing  Hall  of  the  main  SLSC  building,  19  YES  teens  gather. 
Promptly  at  9:00,  George  has  them  start  writing  in  their  journals  for  10  minutes. 
Music  is  playing  over  the  drilling  from  construction  workers  putting  the  finishing 
touches  on  the  new  exhibit  hall.  The  teens  are  fortunate  to  have  the  otherwise 
empty  hall  for  their  summer  activities,  even  though  the  large  space  is 
occasionally  shared  with  the  Exhibit  Lab  teens  and  with  children  in  the  SLSC’s 
summer  camp. 

With  journals  remaining  out,  George  turns  off  the  music  and  spells  out 
brainteasers  for  the  teens  to  add  to  their  journals.  (They  have  adapted  to  the 
space  and  the  lack  of  white  boards.)  They  write  “MEREPEAT”  and  “COTAXME”, 
and  George  asks  them  what  the  words  say.  The  teens  discuss  possible 
meanings  at  their  tables,  while  three  summer  interns  look  on.  George  walks 
among  the  tables,  encouraging  a  few  teens  that  seem  to  have  given  up  quickly. 
He  finally  gives  them  the  first  answer  (repeat  after  me),  and  then  several  say  they 
have  the  second  (income  tax).  After  additional  brainteasers,  now  that  they 
understand  the  concept,  they  stop  and  return  their  journals  to  the  milk  crate  that 
serves  as  their  storage. 

Unlike  the  teens  in  Summertime  Science  who  work  with  a  different  community 
group  each  day,  the  astronomy  group  sees  the  same  children  from  the  same 
community  centers  throughout  the  summer.  Today  the  teens  prepare  for  thirty  of 
these  younger  children,  which  they  will  divide  into  three  rotations.  Teens  are 
divided  into  four  groups,  one  to  lead  each  rotation  with  the  younger  children  and 
one  to  go  into  the  Planetarium  to  lead  activities  with  visitors. 

At  1 0:00,  the  teens  wrap  up  preparations  for  the 
morning  groups  as  George  enters  the  large  hall  saying 
“show  time!”  The  children  from  the  one  community 
organization  follow  him  in.  Half  sit  with  YES  teens  at 
tables  for  “What  is  Life”  activities,  and  half  sit  on  the 
floor  with  their  YES  teen  leaders  for  a  “Lunar  Lander” 
activity  from  NASA  curriculum.  A  third  group  of  YES 
teens  wait  for  their  group  of  children  to  arrive  by  van 
from  their  community  center  so  they  can  lead  the  Life 
Science  Lab  rotation,  and  a  fourth  group  of  YES  teens 
heads  outside  to  Forest  Park  for  outdoor  inquiry 
activities  with  one  of  the  interns. 

After  the  community  groups  leave  and  the  teens  finish  a  lunch  break,  the 
teens  leading  the  three  morning  rotations  debrief  and  plan  for  the  next  day’s 
activities.  At  the  same  time,  about  12:15  PM,  the  fourth  group  of  five  teens  goes 
to  the  Planetarium  with  an  intern  to  lead  activities  with  visitors.  One  teen  helps  a 
SLSC  volunteer  facilitate  Mission  Control  activities.  Another  uses  an  iPad  with  a. 
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Mars  rover  simulation/game  app  to  engage  visitors  in  the  Planetarium  lobby 
and  encourage  them  to  see  the  actual  rover  tucked  back  beside  the  shop 

The  three  remaining  teens  take  over  space 
in  the  hallway  between  Mission  Control  and  the 
Planetarium  lobby  with  demonstrations  they 
developed  themselves.  One  stands  along  the 
east  wall  with  two  suitcases,  one  as  it  would 
weigh  on  earth  and  the  other  demonstrating  its 
weight  on  the  moon.  The  other  two  teens 
demonstrate  earth/moon/sun  relationships. 


“I’m  the  moon,  ”  says  Delaney  holding  up  a  tennis  ball  as  a  family  starts  down  the 
hallway.  “I’m  the  earth,  ” 
shouts  Kelly.  Without  missing  a  beat,  Delaney 
says,  “We  need  a  sun!”  as  she  holds  up  a  yellow 
balloon  and  looks  right  at  the  family.  The  little 
boy  holds  the  balloon  as  the  two  YES  teens 
demonstrate  eclipses.  Once  finished,  Delaney 
says,  “Enjoy  your  visit!”  As  the  family  leaves, 
the  mother  says  to  the  little  boy,  “You  were  the 
sun!”  He  replies,  “Yes!”  She  then  says,  “You  are 
our  son.  ” 


At  12:45,  the  teen  from  Mission  Control  and  the 

teen  from  the  rover  meet  in  the  lobby  to  help  visitors 
make  paper  airplanes  and  rockets.  With  supplies  on  a 
cart,  the  teens  start  to  get  organized  as  a  large  group 
from  a  YMCA  summer  camp  walk  up  and  ask  to  make 
paper  airplanes.  Darian  and  two  YMCA  counselors  help 
the  group  of  10  elementary  school  aged  children  make 
the  planes.  Within  a  few  minutes  the  children  are  flying 
planes  across  the  lobby  before  the  counselors  can  get 
them  to  write  their  names  on  the  planes  and  move  to  a 
safe  flying  space. 


Exhibit  Lab 

New  in  Summer  2012  was  the  Exhibit  Lab  component,  which  was  designed  to 
bring  exhibit  developers,  production  staff,  and  evaluators  from  the  SLSC  together 
with  teens  to  design  and  prototype  exhibits.  The  teens  were  divided  into  four 
groups,  each  led  by  three  to  four  interns,  many  of  whom  were  former  YES  teens. 
Teens  began  working  in  four  classrooms  at  Compton-Drew  Middle  School,  next 
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door  to  the  science  center’s  main  building.  Once  the  school  had  to  focus  on 
preparing  the  building  for  the  school  year,  the  Exhibit  Lab  teens  moved  in  to  the 
large  exhibit  hall  to  share  space  with  the  Astronomy  component.  The  following 
story  picks  up  as  the  teens  actually  begin  to  test  their  prototypes  in  the  Human 
Adventure  Gallery,  and  follows  one  group  of  teens  led  by  Cheryl,  Raymond,  and 
Jim. 


Exhibit  Lab  Component  Vignette 

After  a  morning  of  last  minute  preparations,  at  1:10  PM  on  this  typically 
crowded  science  center  summer  Tuesday,  Andrea  tells  the  group  of  YES  teens 
to  get  ready  to  move  their  prototypes  onto  the  gallery  floor.  Soon  the  four  small 
groups  move  their  carts  into  the  gallery  and  prepare  for  visitors,  consisting 
primarily  of  families  with  children  and  a  few  summer  camp  groups. 

As  visitors  walk  into  the  Human  Adventure  gallery,  they  are  greeted  by  three 
teens  standing  in  front  of  their  prototype  of  a  car  simulator.  They  greet  visitors 
and  ask  ]f  they  would  like  to  try  their  demonstration. 

One  teen  has  a  father  hit  his  fist  on  the  cart  in  a 
specific,  yet  complex  pattern,  and  then  explains  the 
brain’s  response  to  multitasking.  She  concludes  by 
pointing  to  the  dangers  of  texting  while  driving. 


A  basketball  exhibit  prototype  attracts  a  brother  and 
sister  who  want  to  try  for  a  basket. 


At  a  music  exhibit  prototype,  parents  and  children 
listen  to  music  with  headphones  then  discuss  their 
experience  with  the  YES  teens  as  they  explain  the 
relationship  between  music,  emotions,  and  the  brain. 


At  a  puzzle  exhibit  prototype,  visitors  walk  up  to 
try  their  hand  at  several  puzzles. 


Throughout  the  prototyping  process,  the  teens  ask  for  visitor  feedback  using 
surveys  they  created  with  the  help  of  the  science  center’s  evaluation  staff. 
Andrea,  Raymond  and  Jim  walk  from  exhibit  to  exhibit  to  offer  support,  but  play  a 
minimal  role  and  let  the  teens  take  the  lead. 
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Appendix  B  -  Evaluation  Activities 


Evaluation  Activities  to  Date 

2010 

•  December  -  YES  alumnae/alumni  survey  at  event 

•  December  -  February  201 1  -  Evaluation  team  planning 
2011 

•  January  -  November  -  Information  interviews  with  stakeholders 

•  January  -  December  -  Observations  of  staff  meetings 

•  January  -  December  -  Meetings  with  PI 

•  January  -  March  -  Observations  of  staff  training  &  PD 

•  February  -  April  -  Observations  of  spring  YES  Program 

•  April  -  Spring  YES  Teen  survey 

•  June  -  July  -  Observations  of  summer  YES  Program 

•  June  -  Summer  YES  Teen  survey 

•  March  -  August  -  IRB  application,  meetings,  and  approval 

•  October  -  Observations  of  fall  YES  Program 

•  November  -  Fall  YES  Teen  survey 
2012 

•  January  -  April  -  Observations  of  spring  YES  Program 

•  January  -  December  -  Observations  of  staff  meetings 

•  January  -  March  -  Information  interviews  with  stakeholders 

•  January  -  December  -  Meetings  with  PI 

•  April  -  Spring  YES  Teen  survey  (with  TOSRA2) 

•  April  -  Observations  of  staff  PD 

•  May  -  Focus  group  with  community  partners 

•  June  -  July  -  Observations  of  summer  YES  program 

•  July  -  Summer  YES  Teen  survey  (with  TOSRA2) 

•  July  -  Focus  groups  with  teens  and  staff 

•  September  -  Information  interviews  with  managers 

•  October-November  -  Observations  of  fall  YES  Program 

•  November  -  Fall  YES  Teen  survey 
2013 

•  January  -  April  -  Observations  of  spring  YES  Program 

•  January  -  May  -  Observations  of  staff  meetings 

•  January  -  April  -  Meetings  with  PI 

•  February  -  March  -  Alumni  online  survey  pilot 

•  April  -  Spring  YES  Teen  survey  (with  TOSRA2) 

•  April  -  Participant  observation  of  national  partners  meeting 

•  July  -  Summer  YES  Teen  survey  (with  TOSRA2) 
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Evaluation  Reports  Submitted 

•  March  201 1  -  Evaluation  Progress  Report 

•  April  201 1  -  Summary  of  Spring  YES  Teen  Surveys 

•  August  201 1  -  Summary  of  Summer  YES  Teen  Surveys 

•  September  2011  -  Evaluation  Progress  Report 

•  November  201 1  -  Summary  of  Fall  YES  Teen  Surveys 

•  April  2012  -  Summary  of  Spring  YES  Teen  Surveys 

•  August  201 2  -  Summary  of  Summer  YES  Teen  Surveys 

•  November  2012  -  Second  Annual  Evaluation  Report 

•  Fall  2012  -  Summary  of  Fall  YES  Teen  Surveys 

•  April  2013  -  Summary  of  Spring  YES  Teen  Surveys 

•  April  201 3  -  Summary  of  YES  Alumni  Surveys 

•  August  2013  -  YES  Program  High  School  Graduation  Rates 

•  October  201 3  -  Dissemination  Plan 

•  October  201 3  -  Research  Agenda 

•  October  201 3  -  Summative  Evaluation  Report 
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Appendix  C  -  Data  Sources 


This  appendix  discusses  the  sources  for  data  presented  in  this  report.  Data 
sources  include  surveys,  observations,  interviews  (in-depth  interviews,  interactive 
interviews,  and  focus  groups),  and  program  records. 

Evaluators  used  two  types  of  surveys  to  collect  data  from  YES  Teens 
participating  in  the  program.  Both  types  of  surveys  involved  population  samples. 
The  population  number  for  each  survey  is  the  number  of  participating  YES  Teens 
for  the  semester  in  which  the  survey  was  administered.  We  defined  participating 
teens  as  those  who  attended  at  least  twice  during  the  semester.  Printed  surveys 
were  distributed  to  respondents  during  YES  Program  Learning  Labs.  Table  C.1 
shows  the  response  rate  for  each  survey. 

Teen  Surveys  were  developed  by  the  External  Evaluator.  Several  items  were 
consistent  from  semester  to  semester  to  allow  comparison  and  other  items 
provided  snapshots  about  specific  topics  relevant  to  ongoing  evaluation  issues 
and  concerns. 

The  Test  of  Science  Related  Attitudes  (TOSRA)  survey,  a  standardized 
instrument,  was  developed  by  Fraser  (1981)  and  used  internationally.  The 
instrument  was  designed  to  measure  secondary  science  students’  attitudes 
toward  science,  using  70  statements  with  seven  subscales  and  a  5-point  Lickert 
Scale  (strongly  agree,  agree,  not  sure,  disagree,  and  strongly  disagree).  TOSRA 
has  been  used  with  youth  around  the  world,  and  has  been  shown  to  be  valid  and 
reliable  for  American  teens.  A  modified  version  (TOSRA2)  developed  by 
Ledbetter  and  Nix  (2002)  was  used  in  this  study,  consisting  of  35  pre-test  items 
and  35  post-test  items  with  negatively  and  positively  phrased  items  balanced  on 
each  test. 

The  seven  subscales  are  as  follows: 

•  Social  Implications  of  Science  -  Do  youth  recognize  the  benefits  and 
drawbacks  of  scientific  advances  to  society? 

•  Normality  of  Scientists  -  Do  youth  see  scientists  as  real  people  rather  than 
media-produced  stereotypes? 

•  Attitude  toward  Scientific  Inquiry  -  Do  youth  view  experimentation  and 
inquiry  as  a  way  to  gain  understanding  of  the  natural  world? 

•  Adoption  of  Scientific  Attitudes  -  Have  youth  adopted  the  attitudes  of 
scientists,  such  as  open-mindedness  and  self-assessment? 

•  Enjoyment  of  Science  Lessons  -  To  what  degree  do  youth  enjoy  their 
lessons  in  school  science  classes? 

•  Leisure  Interest  in  Science  -  To  what  degree  are  youth  interested  in 
science  out  of  school,  and  outside  of  the  YES  Program? 

•  Career  Interest  in  Science  -  Do  youth  have  an  interest  in  pursuing  a 
science  related  career? 
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This  science  attitude  survey  was  administered  four  times  during  the  project. 

The  population  for  surveys,  as  shown  in  Table  C.1,  was  all  teens  attending 
the  program  at  least  once.  A  few  teens  that  attended  only  one  time  were  present 
on  days  when  the  survey  was  conducted.  Readers  should  note  that  this 
population  definition  is  different  from  that  used  to  figure  attendance.  In  figuring 
attendance,  participating  teens  were  defined  as  those  who  attended  at  least  two 
times. 


Table  C.1.  Surveys 


Surveys 

Name  of  Data  Set 

Respondent 

Group(s) 

Population 

N 

Respondent 

N 

Response 

Rate  (%) 

Date 

Teen  Surveys 

Teen  Survey  Data- 
Spring  201 1 

Semester 

Participants 

246 

186 

75.6 

Spring 

2011 

Teen  Survey  Data- 
Summer  201 1 

Semester 

Participants 

280 

220 

78.5 

Summer 

2011 

Teen  Survey  Data- 
Fall  2011 

Semester 

Participants 

176 

122 

69.3 

Fall  2011 

Teen  Survey  Data- 
Spring  2012 

Semester 

Participants 

216 

109 

50.4 

Spring 

2012 

Teen  Survey  Data- 
Summer  2012 

Semester 

Participants 

251 

194 

77.3 

Summer 

2012 

Teen  Survey  Data- 
Fall  2012 

Semester 

Participants 

167 

111 

66.5 

Fall  2012 

Teen  Survey  Data- 
Spring  2013 

Semester 

Participants 

162 

122 

75.3 

Spring 

2013 

Teen  Survey  Data- 
Summer  2013 

Semester 

Participants 

195 

124 

63.6 

Summer 

2013 

Test  of  Science 
Related 

Attitudes 

(TOSRA) 

TOSRA,  April  2012 

Semester 

Participants 

216 

128 

59.2 

April,  2012 

TOSRA,  July  2012 

Semester 

Participants 

251 

194 

77.3 

July,  2012 

TOSRA,  May  2013 

Semester 

Participants 

162 

110 

67.9 

May,  2013 

TOSRA,  July  2013 

Semester 

Participants 

195 

115 

59.0 

July,  2013 

TOSRA,  Matched 
April  2012  with  July 
2013 

Matched 

Pairs 

310 

44 

14.2 

July,  2013 

Observations  were  conducted  by  the  evaluation  team  and  by  the 
documenters  hired  for  the  Summer  201 1  program.  Only  those  by  the  evaluation 
team  (KK  and  CT)  were  included  in  analysis  for  this  report,  as  reported  in  Table 


C.2. 
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Table  C.2.  Observations 


Observations 

Name  of  Data  Set 

Respondent 

Group(s) 

Respondent 

N 

Observer 

Date 

Evaluator 
Observations  of 
Staff  Meetings 
and  PD 

Staff  Meeting 

032912 

Staff  Members 

11 

KK 

3/29/12 

Community  Partner 
Meeting  050812 

Staff  Members 
and  Community 
Partners 

27 

KK 

5/8/12 

Staff  Meeting 

030212 

Stakeholder  and 
Staff  Members 

16 

KK 

3/2/12 

Staff  Professional 
Development 

042512 

Staff  Members 

8 

KK 

4/25/12 

Staff  Storyboarding 
Meeting  051412 

Staff  Members 
and 

Stakeholders 

15 

KK  & 
CT 

5/14/12 

Staff  Meeting 

062012 

Staff  Members 

55 

KK 

6/20/12 

Staff  Meeting 

090712 

Staff  Members 
and 

Stakeholders 

13 

KK 

9/7/12 

Staff  Meeting 

121412 

Staff  Members 

11 

KK 

12/14/12 

Staff  Meeting 

012913 

Staff  Members 

16 

KK 

1/29/13 

Staff  Meeting 

030113 

Staff  Members 

8 

KK 

3/1/13 

Evaluator 
Observations  of 
Learning  Labs 

College  Prep 

Learning  Lab 

031712 

YES  Teens, 

Staff  Members, 
Interns 

17 

CT 

3/17/12 

Astronomy  Learning 
Lab  031712 

YES  Teens  and 
Staff  Members 

17 

CT 

3/17/12 

Robotics  Learning 

Lab  Summer  2012 
062112  &  062212 

YES  Teens  and 
Staff  Members 

18 

CT 

6/20- 

22/12 

Astronomy  Learning 
Lab  062612 

YES  Teens  and 
Staff  Members 

-18 

KK 

6/26/12 

Summertime 

Science  Learning 

Lab  062012 

YES  Teens  and 
Community 

Group  Youth 

27 

KK 

6/20/12 

In-depth  interviews,  interactive  interviews,  and  focus  group  interviews  were 
conducted  by  evaluation  team  members  and  were  transcribed  for  analysis. 


Klein  Consulting 


69 


A162 


Table  C.3.  Interviews 


Interviews 

Name  of  Data  Set 

Respondent  Group(s) 

Respondent  N 

Interviewer(s) 

In-depth 

Interviews 

Principal  Investigator 
Interview  01/04/11 

Staff  Member-Education  VP 
and  Grant  PI 

1 

KK 

Office  of  Naval 

Research  Program 
Officer  1/25/11 

Stakeholder-Funder 

1 

KK 

Manager  Interview 
03/18/11 

Staff  Member-Manager 

1 

KK,  CT 

St.  Louis  Science 
Center  Administrator 
04/06/1 1 

Stakeholder-Institutional 

Administration 

1 

KK 

St.  Louis  Science 
Center  Board  of 
Trustees  Member 
11/15/2011 

Stakeholder-Board  of 
Trustees  Member 

1 

KK 

St.  Louis  Science 
Center  President 
03/06/12 

Stakeholder-Institutional 

Administration 

1 

KK 

Staff  Member 

Interview  03/21/11 

Program  Staff  Member 

1 

KK,  CT 

Staff  Member 

Interview  07/19/11 

Program  Staff  Member 

1 

KK,  CT 

Manager  Interview 
07/19/11 

Staff  Member-Manager 

1 

KK,  CT 

Manager  Interview 
09/05/12 

Staff  Member-Manager 

1 

KK,  CT 

Manager  Interview 
09/15/12 

Staff  Member-Manager 

1 

KK,  CT 

Interactive 

Interviews 

Interactive  Interviews 
7/23-24/13 

YES  Teens 

10 

KK 

Focus  Groups 

Community  Partner 
Focus  Group  05/09/12 

Community  Partners 

8 

CT,  KK 

Teen  Focus  Group 
07/17/12 

YES  Teens 

9 

CT,  KK 

Teen  Focus  Group 
07/18/12 

YES  Teens 

10 

CT,  KK 

Summer  Staff  Focus 
Group  08/02/12 

Summer  Staff  Members 

10 

CT,  KK 

Senior  Educator 

Focus  Group  08/21/12 

Senior  Educators 

7 

CT,  KK 
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Program  records  were  collected  from  staff  members  by  the  evaluators. 
Records  included  attendance  data,  demographic  and  other  details  on  individual 
teens,  and  documents  shared.  The  Career/College  Readiness  Interview  was 
conducted  by  a  Senior  Educator  by  phone  with  recent  graduating  seniors. 


Table  C.4.  Program  Records 


Program 

Records 

Name  of  Data 
Set 

Respondent 

Group(s) 

Population 

N 

Respondent 

N 

Response 

Rate 

Date 

Career/College 

Readiness 

Interview 

Career- 

College 

Readiness 

Plan 

YES  Teens- 
Seniors  2012 

62 

57 

91.9% 

Spring 

2012 

Senior  College 
Data  2013 

YES  Teens- 
Seniors  2013 

57 

46 

80.7% 

Spring 

2013 

ALL  FORMER 
TEENS  copy 

YES  Teens 
through  2010 

627 

4/21/12 

2010  Current 
Teen 

Information 

YES  Teens  Full 
Roster 

235 

6/8/1 1 

Teen  Database 

2011  Current 
Teen 

Information 

YES  Teens  Full 
Roster 

333 

7/27/12 

2012  Current 
Teen 

Information 

YES  Teens  Full 
Roster 

262 

8/29/12 

Attendance 

YES 

Attendance 

Fall  2010 

Teens  Assigned 
to  Learning  Labs 

218 

12/13/10 

YES  Program 
Summer  2010 
Attendance 

Teens  Assigned 
to  Learning  Labs 

209 

10/7/10 

YES 

Attendance 
Spring  201 1 

Teens  Assigned 
to  Learning  Labs 

234 

5/4/1 1 

YES 

Attendance 
Summer  201 1 

Teens  Assigned 
to  Learning  Labs 

301 

8/15/11 

YES 

Attendance 

Fall  2011 

Teens  Assigned 
to  Learning  Labs 

193 

1/2/12 

YES 

Attendance 
Spring  2012 

Returning  Teens 
Assigned  to 
Learning  Labs 

252 

5/3/12 

NEW  TEEN 

YES 

Attendance 
Spring  2012 

Teens  Assigned 
to  Learning  Labs 

63 

9/27/12 
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YES 

Attendance 
Summer  2012 

Teens  Assigned 
to  Learning  Labs 

252 

8/2/12 

YES 

Attendance 

Fall  2012 

Teens  Assigned 
to  Learning  Labs 

167 

12/8/12 

YES 

Attendance 
Spring  2013 

Teens  Assigned 
to  Learning  Labs 

162 

5/18/13 

NEW  TEEN 

YES 

Attendance 
Spring  2013 

Teens  Assigned 
to  Learning  Labs 

49 

6/4/13 

YES 

Attendance 
Summer  2013 

Teens  Assigned 
to  Learning  Labs 

195 

8/5/13 
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Appendix  D  -  T0SRA2  Analysis  Results 


TOSRA  was  chosen  to  measure  changes  in  YES  Teen  attitudes  because  it 
was  found  to  be  a  reliable  and  valid  instrument  for  teenagers  in  other  settings. 
The  TOSRA2  version  was  chosen  due  to  the  shortened  time  to  complete  the 
questions.  Inconsistencies  were  found  with  TOSRA2;  however,  it  is  not  believed 
that  these  inconsistencies  affect  the  findings  from  the  matched  pairs  presented  in 
this  report.  Questions  about  uses  of  the  test  for  this  urban,  American,  21st  century 
population  are  covered  below. 

TOSRA2  scores  were  recorded  in  an  Excel  spreadsheet  and  used  in  an 
SPSS  program  for  analysis.  Scores  for  each  of  the  seven  categories  and  a  total 
were  calculated  for  each  respondent  on  each  test.  Analysis  was  conducted  using 
descriptive  statistics,  two-tailed  t-test,  one-way  ANOVA,  and  Pearson  Correlation 
on  matched  pairs.  The  matched  pairs  used  for  this  analysis  were  scores  from 
teens  taking  both  the  2012  spring  pre-test  and  the  2013  summer  post-test,  which 
provided  the  greatest  time  between  tests. 

The  findings  include  only  those  results  from  the  matched  pairs. 

Matched  Pair  Results 

Analysis  of  the  matched  pairs  found  correlation  coefficients  to  be  significant  in 
six  categories  and  on  the  total  scores.  Adoption  of  Scientific  Attitudes  shows  a 
trend  toward  positive  gain  with  exact  significance  at  p  <  .056.  Positive  gains  from 
pre  to  post-test  were  found  for  five  of  the  categories  and  for  the  total  scores. 
Negative  gains  from  pre  to  post-test  were  found  for  two  of  the  categories 
(Attitude  Toward  Inquiry  and  Career  Interest).  Two-sided  paired  t-tests  found 
significant  differences  (p  <  0.05)  in  five  of  the  categories  and  the  total  scores. 

The  distribution  of  means  for  the  seven  categories  (subscales)  on  the  pre¬ 
test,  as  seen  in  Table  18  and  Figure  16  on  pages  37-38,  follows  a  pattern  similar 
to  that  found  by  Mountz  (2006)  with  Leisure  Interest  receiving  the  lowest  scores 
and  Attitude  Toward  Inquiry  receiving  the  highest  pre-test  scores. 

Gender 

We  found  no  significant  difference  between  male  and  female  respondents’ 
gain  scores  (i.e.  difference  from  pre  to  post-test)  on  all  seven  TOSRA  categories 
and  on  total  scores.  However,  we  did  note  that  males  tended  to  answer  lower  on 
the  five  point  scale  than  females  in  all  categories,  except  Career  Interest  on  the 
pre-test  (not  significant  using  a  one-way  ANOVA).  The  subscale  scores  on  the 
post-test  for  females  on  Attitude  toward  Scientific  Inquiry  were  significantly  higher 
than  scores  for  males  (mean  of  19.80  for  females,  16.79  for  males,  with  one-way 
ANOVA  p<  0.01). 
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Grade  in  School  as  an  Indication  of  Maturity 

Using  grade  level  (year  of  high  school  graduation)  as  an  indication  of 
maturity,  no  significant  differences  were  found  for  all  TOSRA  categories  and  total 
scores  for  the  2012  pre-test  and  2013  post-test  using  a  one-way  ANOVA.  All  total 
scores  increased  by  grade  level  except  for  a  decrease  in  the  one  9th  grade  teen 
as  seen  in  Figure  D.1  and  Tables  D.1  and  D.2.  Total  scores  by  fall  2013  grade 
level  are  shown  in  Table  D.1.  Changes  in  mean  scores  are  shown  in  Table  D.2 
with  negative  changes  shaded. 
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Figure  D.  1.  Average  total  scores  for  the  spring  2012  pre-test  and  summer  2013  post-test 
TOSRA2  by  grade  in  school  in  fall  2013. 


Table  D.1.  Average  total  scores  by  grade  in  school  in  fall  2013. 


9tn 

10tn 

11th 

12tn 

H.S. 

Grad 

Total 

2012  pre-test  average  score 

125 

117 

121 

113 

99 

118 

2013  post-test  average  score 

107 

125 

126 

116 

124 

124 

Number  in  sample  (N) 

1 

8 

26 

7 

2 

44 

Table  D.2.  Change  in  mean  scores  by  category  and  grade  in  school  in  fall  2013. 


S 

N 

1 

A 

E 

L 

C 

Total 

9th  Grade 

-1.00 

0.00 

-5.00 

-1.00 

-2.00 

-6.00 

-3.00 

-18.00 

10th  Grade 

1.25 

3.63 

-1.50 

2.38 

1.38 

2.00 

-0.88 

8.25 

11th  Grade 

1.04 

2.65 

-0.69 

0.77 

0.80 

1.62 

-1.23 

4.96 

12th  Grade 

0.71 

2.71 

-2.29 

-0.43 

1.00 

0.86 

0.29 

2.86 

Graduate 

5.50 

4.50 

-2.50 

-0.50 

7.50 

9.00 

1.50 

25.00 
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Length  of  Time  in  YES 

A  significant  difference  (p  <  0.05)  was  found  between  length  of  time  in  the 
program  and  Attitude  toward  Inquiry  using  the  Pearson  Correlation.  In  correlating 
mean  scores  for  each  category  with  how  long  the  teen  had  been  in  the  program 
at  the  time  of  the  post-test  (Table  D.3),  results  showed  that  the  longer  teens  were 
in  the  YES  Program,  the  lower  their  mean  score  on  Attitude  toward  Inquiry.  This 
is  consistent  with  the  mean  score  decrease  from  pre  to  post  test  overall.  Thus, 
the  longer  a  teen  is  in  the  YES  Program,  the  less  likely  she  or  he  is  to  view 
experimentation  and  inquiry  as  a  way  to  gain  understanding  of  the  natural  world. 

Table  D.3.  Correlation  results  of  TOSRA  categories  and  length  of  time  in  the  YES 
Program 


Length  of 
Time  in  YES 
Program 

Pearson  Correlation 

-.026 

GainS 

Sig.  (2-tailed) 

.867 

N 

44 

Pearson  Correlation 

-.257 

GainN 

Sig.  (2-tailed) 

.092 

N 

44 

Pearson  Correlation 

.309* 

Gainl 

Sig.  (2-tailed) 

.041 

N 

44 

Pearson  Correlation 

.154 

GainA 

Sig.  (2-tailed) 

.318 

N 

44 

Pearson  Correlation 

-.229 

GainE 

Sig.  (2-tailed) 

.135 

N 

44 

Pearson  Correlation 

-.228 

GainL 

Sig.  (2-tailed) 

.137 

N 

44 

Pearson  Correlation 

-.189 

GainC 

Sig.  (2-tailed) 

.219 

N 

44 

Pearson  Correlation 

-.126 

GainTotal 

Sig.  (2-tailed) 

.413 

N 

44 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed) 

*  Correlation  is  significant  at  the  0.05  level  (2-tailed) 

Analyses  took  into  account  a  number  of  factors  that  could  affect  attitudes: 
gender,  ethnicity,  age/maturity,  and  time  in  the  YES  Program.  The  ethnic 
diversity  of  the  matched  pairs  was  similar  to  that  of  the  YES  Program,  and  did  not 
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have  enough  variation  to  draw  conclusions  about  differences  in  attitudes  based 
on  ethnicity.  No  significant  differences  in  attitude  were  found  by  gender  on  gain 
scores,  though  males  tended  to  rate  items  lower  in  general  than  females  and 
females  rated  Attitude  Toward  Inquiry  significantly  higher  than  males  on  the  post¬ 
test.  A  significant  difference  was  found  between  length  of  time  in  the  program 
and  Attitude  toward  Inquiry  suggesting  the  longer  a  teen  was  in  the  YES 
Program,  the  less  likely  she  or  he  was  to  view  experimentation  and  inquiry  as  a 
way  to  gain  understanding  of  the  natural  world.  No  significant  differences  were 
found  based  on  age  as  an  indication  of  maturity,  though  the  9th  grader  rated 
categories  lower  on  the  post-test  and  the  two  high  school  graduates  rated  all 
except  two  categories  higher  on  the  post-test  than  on  the  pre-test. 

The  TOSRA  was  chosen  to  measure  YES  Teen  attitudes  toward  science  and 
scientists  because  it  was  found  to  be  reliable  and  valid  by  researchers  in  the 
United  States  and  elsewhere  in  the  1980s,  1990s  and  early  2000s.  Results  of 
TOSRA2  with  YES  Teens  called  into  question  the  construct  validity  of  TOSRA2 
with  urban  teens  in  this  time  period.  Did  the  test  actually  measure  what  it  was 
supposed  to  measure? 

The  economic  climate  has  shifted  considerably  since  Fraser  first  developed 
the  TOSRA,  and  some  questions  referred  to  money  spent  on  science  or 
education.  Technology  has  changed  and  questions  did  not  include  attitudes 
toward  technology  or  the  use  of  technology  in  science.  Of  larger  concern  is  the 
shift  in  scientific  practices.  Where  scientists  once  worked  independently,  more 
and  more  work  collaboratively.  YES  Teens  are  encouraged  to  see  science  as  a 
field  for  collaboration,  and  scientists  as  people  who  enjoy  working  together. 

With  construct  validity  called  into  question,  each  category  can  be  considered 
in  light  of  findings  and  construct  validity  questions. 

Social  Implications  of  Science  (S)  was  designed  to  measure  attitudes  toward 
the  societal  benefits  and  drawbacks  of  scientific  advances.  Findings  indicated  a 
significant  increase  in  scores  from  pre-test  to  post-test  for  the  44  matched  pairs 
(means  from  17.05  pre  to  18.23  post,  N=44).  Similar  results  were  found  using 
pre-test  and  post-test  scores  for  all  respondents  of  each  test  (means  from  17.86 
pre  to  18.23  post,  N=133  pre  and  N=1 15  post).  Mean  subscale  scores  increased 
for  matched  pairs  at  all  grade  levels  except  for  the  one  9th  grade  teen.  Findings 
suggest  that  the  YES  Program  leads  to  improved  attitudes  toward  scientific 
advances. 

Normality  of  Scientists  (N).  Teens  may  believe  the  media  stereotypes  about 
scientists  when  they  enter  the  YES  Program,  but  the  Community  STEM  Outreach 
project  was  designed  to  introduce  YES  Teens  to  real  scientists  to  expose  them 
as  real  people.  This  TOSRA  subscale  was  developed  to  measure  changes  in 
attitudes  toward  scientists.  Findings  indicate  a  significant  increase  in  scores  from 
pre-test  to  post-test  for  the  44  matched  pairs  (means  from  15.59  pre  to  18.45 
post),  the  largest  change  of  the  subscales.  Similar  results  were  found  using  pre¬ 
test  and  post-test  scores  for  all  respondents  of  each  test  (means  from  15.74  pre 
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to  18.46  post).  Mean  subscale  scores  increased  for  matched  pairs  at  all  grade 
levels. 

Attitude  to  Scientific  Inquiry  (I).  The  YES  Program  is  grounded  in  inquiry  as  a 
way  to  gain  understanding  of  the  natural  world.  Project  goals  include  increasing 
YES  Teens’  valuing  of  inquiry  and  experimentation,  and  increasing  their  skills  in 
the  process  of  inquiry.  TOSRA  was  designed  to  measure  changes  in  attitudes 
(rather  than  skills).  Results  from  the  TOSRA2  showed  a  decrease  in  total  and 
subscale  scores  for  the  matched  pairs,  and  at  each  grade  level.  Change  differed 
due  to  the  length  of  time  a  teen  was  in  the  program,  with  those  in  YES  longer 
having  more  negative  change  from  pre  to  post.  From  this  it  would  be  easy  to 
conclude  that  after  16  months  or  more  in  the  YES  Program,  teens  were  less 
likely  to  value  the  inquiry  process.  However,  a  closer  look  at  the  statements  on 
TOSRA2  calls  into  question  the  construct  validity  with  this  generation  of  teens. 
For  example,  the  pre-test  asks  teens  to  rate  “I  would  rather  find  out  why 
something  happens  by  doing  an  experiment  than  by  being  told  how  it  works,” 
with  an  average  of  4.1 1  (high  agreement)  on  a  5  point  scale,  N=133.  The  post¬ 
test  says  “I  would  rather  find  out  about  things  by  asking  an  expert  than  by  doing 
an  experiment,”  with  an  average  response  of  3.23  (slight  disagreement),  N=1 15. 
While  these  statements  appear  to  be  parallel  and  consistent  with  previous 
generations  of  scientists,  there  is  a  subtle  difference.  Results  indicate  that  YES 
Teens  would  rather  experiment  and  learn  on  their  own  through  hands-on 
activities,  but  they  are  not  opposed  to  seeking  advice  and  answers  from  experts. 
Several  statements  on  this  subscale  involved  asking  experts  or  teachers  or 
seeking  answers  from  other  sources,  making  it  difficult  to  draw  conclusions  from 
the  findings.  It  could  also  be  that  having  carried  out  their  own  experiments  teens 
appreciated  the  time  and  effort  required  to  reach  sound  conclusions  and  learned 
to  trust  and  rely  on  information  based  on  the  experimentation  of  others.  Gender 
differences,  particularly  the  significant  differences  on  the  post-test,  indicate  that 
girls  had  a  more  positive  attitude  toward  inquiry  than  their  male  peers  after 
participating  in  the  YES  Program. 

Adoption  of  Scientific  Attitudes  (A).  The  TOSRA2  was  designed  to  measure 
whether  teens  adopted  the  attitudes  of  scientists  over  time,  such  as  open- 
mindedness  and  self-assessment.  Results  from  the  matched  pairs  indicated  a 
positive,  though  weak,  change  over  time.  Results  by  grade  level  in  school 
indicated  that  the  larger  group  of  teens  in  grades  10-1 1  showed  positive  change, 
while  those  in  grade  12  and  the  2  recent  graduates  showed  small  negative 
changes.  The  one  9th  grade  teen  showed  a  larger  negative  change. 

Two  pairs  of  statements  for  this  category  raise  questions.  Item  4  on  the  pre¬ 
test  reads:  “I  find  it  boring  to  hear  about  new  ideas”  (with  a  mean  score  of  3.70, 
good  agreement,  N=133).  This  is  paired  with:  “I  enjoy  reading  about  things  which 
disagree  with  my  previous  ideas”  (with  a  mean  score  of  3.1 1 ,  neutral,  N=1 33). 
This  second  statement  would  be  scored  low  if  a  teen  doesn’t  like  to  read  or  if  the 
teen  doesn’t  like  her  or  his  ideas  challenged.  Item  1 1  on  the  pre-test  reads:  “In 
science  experiments,  I  like  to  use  methods  which  I  have  not  tried  before”  (3.83, 
agreement,  N=115).  The  post-test  item  reads:  “I  dislike  repeating  experiments  to 
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check  that  I  got  the  same  results”  (3.04,  neutral,  N=1 15).  While  teens  like  to  use 
new  methods,  perhaps  they  really  don’t  care  whether  or  not  they  repeat 
experiments  since  they  enjoy  doing  them.  Weak  changes  could  be  related  to  the 
wording  of  the  questions  on  this  subscale. 

Enjoyment  of  Science  Lessons  (E)  does  not  relate  directly  to  the  project  or 
the  goals  of  the  YES  Program.  Instead,  it  could  be  an  indicator  of  teen  attitudes 
toward  science.  The  TOSRA  was  designed  to  measure  enjoyment  of  lessons  in 
school  science  classes.  Findings  show  a  significant  positive  difference  in  scores 
from  pre  to  post-test,  with  all  grade  levels  except  the  one  9th  grader  showing 
positive  change.  It  should  be  noted  that  the  TOSRA2  used  the  term  “science 
lessons”  throughout  rather  than  specifying  science  lessons  in  school.  It  is  unclear 
whether  YES  Teens  interpreted  this  as  science  lessons  in  school  or  in  YES. 

Leisure  Interest  in  Science  (L)  is  another  indicator  of  attitudes,  rather  than 
being  directly  related  to  YES  Program  goals.  Are  teens  more  interested  in 
science  in  their  leisure  time  after  an  additional  16  months  in  the  program?  The 
answer  is  yes,  for  all  grade  levels  except  the  one  9th  grader.  The  highest  positive 
change  was  with  the  two  high  school  graduates. 

Career  Interest  in  Science  (C)  The  TOSRA  examines  interest  in  pursuing  a 
science  related  career.  Change  in  scores  was  small  suggesting  a  slight  decrease 
in  interest  in  science  as  a  career.  By  grade  level,  change  was  negative  for 
students  in  grades  9-1 1  at  the  time  of  the  post-test  and  positive  for  those  in  12th 
grade  and  the  two  recent  graduates,  suggesting  more  positive  changes  as  teens 
mature.  Females  scored  all  subscales  higher  than  males  with  the  exception  of 
Career  Interest  on  the  pre-test,  indicating  lower  interest  as  they  entered  the 
program. 

A  closer  look  at  the  statements  rated  on  this  subscale  raises  questions  in 
interpretation  of  results.  For  example,  Item  7  on  the  pre-test,  “I  would  dislike 
being  a  scientist”  (with  a  mean  score  of  3.16,  N=1 33)  is  paired  with  item  7  on  the 
post-test,  “I  would  like  a  career  teaching  science”  (2.41,  N=1 1 5).  These  are  not 
parallel  for  YES  Teens  since  teaching  science  is  viewed  as  different  from  being  a 
scientist.  Another  example  is  item  35.  Pre-test  states,  “A  career  in  science  would 
be  dull  and  boring”  (3.47,  disagreement,  N=1 33),  while  the  post-test  states:  “I 
would  like  a  career  as  a  scientist”  (2.93,  neutral,  N=1 15).  Perhaps  the  teens  are 
willing  to  say  a  career  in  science  would  be  interesting,  while  at  the  same  time  are 
not  willing  to  commit  to  a  personal  interest  in  becoming  a  scientist. 
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Community  STEM  Outreach 
Proposed  Research  Agenda 

Submitted  October  2013  by  Klein  Consulting1 


INTRODUCTION 

The  Saint  Louis  Science  Center  (SLSC),  in  its  August  4,  2010  proposal  to  the 
Office  of  Naval  Research  (ONR),  agreed  to  “identify  research  questions  related 
to  the  Community  STEM  Outreach  project,  and  create  strategies  for  moving 
forward  with  that  research,  including  seeking  additional  funding  for  such 
research.”  In  December  201 1 ,  the  SLSC  contracted  with  Klein  Consulting  to 
develop  a  strategic  plan  for  additional  research.  This  document  outlines  the 
research  questions,  potential  strategies,  and  efforts  to  date  to  seek  additional 
funding  for  research. 

Research  questions  emerged  from  the  evaluation  of  the  Community  STEM 
Outreach  project  and  from  a  synthesis  of  previous  evaluation  reports  from  the 
Youth  Exploring  Science  (YES)  Program. 

As  the  external  evaluator  on  the  Community  STEM  Outreach  project,  Klein 
Consulting  was  able  to  draw  on  evaluation  results  to  create  this  strategic  plan. 
This  document  represents  the  work  and  views  of  Klein  Consulting.  Any  opinions, 
findings,  conclusions,  or  recommendations  expressed  in  this  document  are  those 
of  the  authors  and  do  not  necessarily  reflect  the  views  of  the  SLSC  or  the  ONR. 


RESEARCH  VERSUS  EVALUATION 

Evaluation  was  a  key  piece  in  the  Community  STEM  Outreach  project.  Like 
most  program  evaluations,  it  focused  on  the  merit  or  worth  of  a  specific  program 
(YES)  within  a  specific  time  (2010-2013),  and  it  informed  decisions  of  one  or 
more  stakeholders.  This  is  one  defining  feature  of  evaluation,  and  is  one  way  that 
evaluation  is  different  from  research  (Guba  &  Lincoln,  1989).  As  is  often  stated, 
evaluation  is  designed  to  improve  something,  while  research  is  design  to  prove 
something.  The  Community  STEM  Outreach  project  evaluation  was  designed  to 
improve  the  YES  Program  and  report  successes  (or  lack  of  success)  to  the 
funder,  the  ONR.  The  research  proposed  in  this  report  is  intended  to  provide  the 
field  of  informal  science  education  with  evidence  of  the  impacts  and  value  of 
youth  STEM  education  programs  in  informal  learning  environments  in  general. 
While  some  research  questions  may  focus  on  the  YES  Program,  they  can  be 
used  with  other  programs  in  other  settings  or  across  programs. 


1  Questions  regarding  this  document  may  be  addressed  to  the  authors:  Christine  Klein,  Principal, 
Klein  Consulting  (ckleinconsulting@gmail.com),  and  Carey  Tisdal,  Director,  Tisdal  Consulting 
(ctisdal@sbcglobal.net). 
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The  evaluation  of  the  Community  STEM  Outreach  project  provides  the 
foundation  for  the  research  questions  to  take  the  work  to  the  next  logical  step. 
Donors  and  funders  often  ask,  “Can  you  prove  it  works?”  or,  “Can  you  prove  my 
money  will  be  well  spent?”  Evaluation  findings  can  address  those  questions  for 
past  programs,  but  can’t  “prove”  a  new  program  will  work.  The  research 
proposed  in  this  report  has  the  potential  to  demonstrate  the  value  of  program 
aspects  across  a  spectrum  of  programs. 


YES  LOGIC  MODEL 

The  YES  Program  logic  model  identified  through  the  Community  STEM 
Outreach  evaluation  provides  a  useful  framework  for  organizing  research 
questions  (See  Figure  1,  taken  from  the  Community  STEM  Outreach:  Second 
Annual  Evaluation  Report,  Klein  &  Tisdal,  2012). 
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Figure  1.  Logic  Model 

The  logic  model  (and  evaluation  results)  suggests  variables  and  research 
questions  focused  on  program  participants,  the  program,  the  outputs  of  the 
program,  and  various  impacts. 


STAKEHOLDER  ISSUES  AND  CONCERNS 

Rationale  for  the  emerging  research  questions  came  from  the  Community 
STEM  Outreach  evaluation.  For  example,  the  Second  Annual  Evaluation  Report 
(Klein  &  Tisdal,  2012)  identified  stakeholder  issues  and  concerns  that  suggested 
questions  related  to  staff  training  and  curriculum  implementation.  Learning  Lab 
observations  suggested  questions  related  to  specific  program  elements. 
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Stakeholders  for  the  YES  Program  and  the  future  research  include  the 
following: 


•  Program  youth 

•  Parents  and  family  of  program  youth 

•  Program  staff  and  administrators 

•  Institution/organization  administrators 

•  Community  partners 

•  Funders 

•  Educators  in  the  field  of  STEM  formal  and  informal  education 

•  Researchers 

Identified  areas  where  viewpoints  varied  among  stakeholders  in  the  2012 
Community  STEM  Outreach  evaluation  report  included  the  following: 

•  Clarification  of  staff  roles  and  responsibilities 

•  Differences  in  views  on  how  STEM  learning  and  teaching  should  occur 
in  out-of-school  time  (OST)  programs 

•  Program  relationships  with  other  parts  of  the  institution 

Areas  of  importance  to  stakeholders  identified  in  the  2012  evaluation  report 
included  these  six  areas: 

•  Intentional  debriefing  and  reflection  of  learning  activities  at  all  levels 
(with  children,  teens,  and  staff) 

•  Additional  focus  in  staff  PD  on  the  facilitation  process  (train-the-trainer) 

•  Curriculum  that  is  well  planned  and  adheres  to  the  intended  learning 
goals  and  context 

•  Ability  of  all  staff  members  to  perform  at  peak  levels  when  the  teen 
population  remains  constant  and  staff  size  shrinks 

•  Absenteeism  of  YES  Teens  and  communication  among  staff  and  teens 
to  maintain  strong  relationships 

•  A  data  tracking  and  management  system  to  better  meet  the  needs  of 
program  staff  and  the  current  and  future  funders  and  researchers 

The  various  viewpoints  and  areas  of  importance  informed  the  research 
questions. 


RESEARCH  QUESTIONS 

The  Community  STEM  Outreach  project  evaluation  addressed  several 
questions,  using  the  YES  Logic  Model  as  an  organizing  framework. 

•  Who  are  the  YES  Teens? 

•  What  happens  in  YES? 

•  What  are  the  program’s  outputs? 

•  How  has  the  program  changed  over  time? 

•  What  are  the  program’s  short-term  impacts? 

•  What  are  the  different  points-of-view  that  stakeholders  have  about  YES? 
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•  What  areas  are  important  to  stakeholders  and  program  success? 

•  What  recommendations  support  program  improvement? 

•  What  are  the  plans  for  the  future  of  the  program? 

Results  are  included  in  the  evaluation  reports.  These  questions  and  the 
evaluation  data  provide  the  foundation  for  research  questions  regarding  impact 
on  youth  and  impact  of  the  dissemination  of  results  on  additional  youth  and 
organizations. 

The  research  questions  listed  in  this  report  focus  on  the  impact  on  the  youth. 
Additional  questions  are  included  based  on  the  use  of  the  documentation  of  the 
YES  Program,  specifically  the  Circles  of  Support  web-based  multimedia  tool,  by 
other  organizations.  While  numerous  other  questions  could  be  asked  related  to 
professional  development  of  staff  and  other  educators,  community  relationship 
building,  leadership  styles,  sustainable  practices,  informal  science  education  of 
children,  STEM  learning  in  out-of-school  time  (OST),  and  others,  those  questions 
are  outside  the  intent  of  the  project  and  this  document. 

It  should  be  noted  that  the  research  questions  provided  here  are  a  beginning 
rather  than  an  exhaustive  list. 

The  YES  Program 

The  Community  STEM  Outreach  project  evaluation  addressed  the  question: 
What  happens  in  the  YES  Program?  Through  vignettes  and  summary  data,  the 
evaluation  gave  a  picture  of  the  program.  For  future  research  spanning  diverse 
programs,  the  question  becomes: 

•  What  key  elements  of  OST  youth  STEM  programs  are  most  effective? 

Program  Outputs 

A  few  studies  report  on  the  nature  and  structure  of  youth  STEM  programs  in 
science  centers  and  other  informal  STEM  learning  environments.  For  example,  in 
a  study  to  provide  an  overview  of  the  numerous  extended  programs  for  youth 
offered  by  science  centers  and  museums  a  decade  after  funding  for  YouthALIVE! 
ended,  and  to  characterize  the  content  and  structure  of  the  programs,  authors 
Cary  Sneider  and  Meg  Burke  (201 1 )  found  1 63  youth  programs  and  identified 
common  characteristics  from  searching  institution  websites.  The  YES  Program 
was  one  of  many  described,  and  was  the  largest. 

For  future  research  emerging  from  the  Community  STEM  Outreach  project 
and  the  YES  Program,  research  questions  regarding  program  outputs  include 
those  focused  on  numbers  and  comparisons: 

•  Are  graduates  of  youth  STEM  programs  in  science  centers  more  likely  than 
their  peers  to  graduate  from  high  school  or  college,  or  enter  a  STEM  career? 
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Short-term  Impacts 

To  study  short-term  impacts  on  youth  we  can  ask  the  following  questions: 

•  Does  intentional  debriefing  of  activities  with  youth  deepen  their  understanding 
of  the  key  concepts  (STEM  or  others)  involved  in  the  activity? 

•  Is  a  youth  program  with  an  emphasis  on  STEM  concepts  more  or  less  likely  to 
produce  more  youth  pursuing  STEM  degrees  or  careers  than  a  youth 
program  focused  on  youth  development? 

•  What  strategies  of  dealing  with  absenteeism  are  most  effective  in  increasing 
levels  of  participation? 

•  What  is  the  difference  in  how  youth  journal  between  those  asked  to  journal  at 
the  beginning  of  a  learning  lab  versus  at  the  end,  and  between  those  asked  to 
respond  to  a  writing  prompt  versus  those  not  given  a  prompt? 

•  How  do  staff  comments  and  responses  to  youth  journal  entries  affect  adult  to 
youth  relationships?  Attendance  rates?  Program  dropout  rates? 

•  How  does  staff  turnover,  particularly  layoffs  and  dismissals,  affect  the  youth  in 
the  program? 

•  Are  youth  outcomes  different  when  there  is  a  stable  staff  versus  when  there  is 
staff  turnover? 

•  How  do  impacts  differ  between  youth  that  are  paid  in  work-based  programs 
and  those  that  are  not  paid? 

•  How  do  impacts  differ  between  youth  programs  that  target  underrepresented 
youth  and  programs  that  target  youth  from  well-represented  populations? 

•  How  do  impacts  differ  between  youth  that  engage  in  long-term  projects  (e.g. 
project-based  learning)  and  those  that  do  shorter  or  no  projects? 

•  What  attributes  of  a  program  do  youth  view  as  welcoming  and  safe?  And, 
what  are  the  differences  in  impacts  based  on  the  attributes  of  a  program? 

•  How  do  short-term  impacts  differ  between  programs  with  long  (years)  versus 
short  (weeks  or  months)  engagement?  And,  between  youth  with  high  and  low 
attendance? 

Long-term  Impacts 

Research  methods  that  include  use  of  comparison  groups  may  be  needed  to 
measure  long-term  impacts  on  the  youth.  While  standard  metrics  may  be  of 
interest  to  some  stakeholders  (e.g.  high  school  graduation  rates,  college 
entrance  rates,  college  graduation  rates,  and  STEM  careers),  additional 
measures  will  provide  valuable  insights  to  additional  questions. 

Questions  related  to  long-term  impacts  include  the  following: 

•  What  are  the  college  entrance  rates  for  youth  participating  in  STEM  education 
programs?  How  do  rates  of  YES  Teens  compare  to  other  teens  from  the  St. 
Louis  area,  and  with  youth  from  other  STEM  education  programs? 

•  How  many  youth  complete  college  and  during  what  time  frame?  How  do  rates 
of  YES  Teens  compare  to  local  and  national  averages,  and  to  outcomes  from 
other  programs? 
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•  What  percent  of  YES  Teens  enter  STEM  and  STEM  related  careers?  How  do 
percents  of  YES  Teens  compare  to  local  and  national  averages,  and  to 
outcomes  from  other  programs? 

•  What  percent  of  YES  Teens  enter  the  field  of  education,  particularly  STEM 
education  careers?  How  do  percents  of  YES  Teens  compare  to  local  and 
national  averages,  and  to  outcomes  from  other  programs? 

•  What  elements  of  the  YES  Program  and  other  programs  have  the  greatest 
long-term  impact? 

•  How  does  the  amount  of  participation,  as  indicated  by  attendance,  affect  long¬ 
term  impact? 

•  What  are  differences  in  long-term  impacts  between  programs  that  focus  on 
STEM  concepts  and  those  that  focus  on  science  process  skills?  Between 
those  with  an  emphasis  on  STEM  and  those  emphasizing  youth 
development? 

Strategic  Impact 

In  addition  to  the  impact  on  the  youth,  research  can  explore  the  strategic 
impact.  What  is  the  long-term  impact  on  the  organizations  involved  and  on  the 
field  of  informal  science  education  as  a  whole?  The  YES  Program  has  been  a 
leader  in  the  science  center  community.  What  is  its  strategic  impact? 

If  the  documentation  of  the  YES  Program  is  disseminated  to  other 
organizations,  questions  emerge  related  to  its  use  and  impact  on  youth  and  staff. 
YES  Program  documentation  includes  the  written  materials,  videos,  and  web- 
based  materials  created  to  describe  the  YES  Program  model.  Research  and 
evaluation  could  clarify  its  impact. 

•  How  are  educators  in  other  organizations  using  the  documentation? 

•  How  effective  are  the  materials  for  staff  at  other  organizations  that  are 
creating  new  youth  STEM  programs? 

•  How  effective  are  the  materials  for  staff  with  organizations  that  want  to 
improve  or  expand  their  existing  youth  STEM  programs? 

•  How  are  educators  and  administrators  at  SLSC  using  the  documentation? 

•  How  do  program  impacts  differ  among  programs  that  used  the  documentation 
for  program  development  and  staff  PD  and  similar  programs  that  did  not? 


STRATEGIES  FOR  MOVING  FORWARD 

At  the  time  of  this  writing,  the  YES  Program  was  in  the  process  of  shifting 
from  a  growth  model  funded  largely  by  federal  grants  to  a  sustainable  model  with 
reduced  numbers  of  youth.  A  Steering  Committee  was  in  place  to  make 
recommendations  to  the  SLSC  Board. 

The  strategies  offered  here  are  based  on  observations,  interviews,  focus 
groups,  meeting  notes,  program  documents,  and  a  synthesis  of  existing  YES 
evaluation  reports.  Strategies  are  designed  to  address  the  research  questions 
listed  above.  If  carried  out,  some  research  may  directly  benefit  the  SLSC; 
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however,  all  potential  research  is  designed  to  benefit  the  fields  of  informal  STEM 
education,  OST  education,  and  youth  development. 

Research  Methodologies 

From  each  research  question  above  could  emerge  a  separate  research 
design  with  specific  methodology  and  data  collection  methods.  While  it  is  beyond 
the  scope  of  this  report  to  outline  each,  methodologies  could  include: 
correlational  research,  quasi-experimental  research,  qualitative  research, 
longitudinal  research,  case  studies,  and  action  research.  Methods  of  data 
collection  could  include:  observations,  interviews,  questionnaires,  focus  groups, 
and  program  document  analysis. 

The  following  examples  of  different  methodologies  address  key  research 
questions. 

Correlational  Research  examines  the  relationships  between  variables,  without 
determining  cause  and  effect.  Examples  include  exploration  of  the  statistical 
relations  between  program  attendance  and  high  school  graduation,  i.e. ,  do  youth 
who  attend  more  often  (or  more  sessions,  or  more  years)  graduate  from  high 
school  at  higher  rates  than  those  who  don’t  (or  than  youth  who  don’t  participate 
in  the  program)?  A  research  design  with  this  methodology  would  involve 
identification  of  program  participants,  collection  of  attendance  data,  collection  of 
graduation  data,  and  calculations  of  correlations  to  identify  patterns. 

Quasi-Experimental  Research  can  be  used  in  situations  like  the  YES  Program 
when  random  assignment  of  participants  to  groups  (i.e.  experimental  design)  is 
not  possible.  For  questions  regarding  YES  and  its  dissemination,  this 
methodology  relies  on  matching  youth  in  the  program  with  youth  in  comparison 
groups  and  statistically  controlling  some  variables.  For  example,  outcomes  could 
be  studied  from  two  different  YES  components  matched  for  years  of  staff 
experience  with  YES.  In  another  example,  college  degrees  (or  majors,  etc.)  for 
YES  alumni//alumnae  could  be  compared  with  those  from  another  STEM 
program,  controlling  for  socioeconomic  status  or  type  of  high  school. 

Qualitative  Research  takes  many  forms  and  focuses  on  gaining  deeper 
understanding  of  human  behavior  in  naturalistic  settings.  An  example  with  youth 
STEM  education  programs  could  include  a  study  of  alumni/alumnae  of  programs 
to  identify  the  most  salient  elements  and  those  that  yield  the  greatest  impact. 
Qualitative  research  can  identify  those  elements  and  yield  hypotheses  that 
additional  research  can  explore. 

Longitudinal  Research  focuses  on  changes  in  individuals  overtime.  With 
youth  programs,  longitudinal  studies  follow  participants  to  examine  outcomes  like 
college  graduation,  career  choice,  and  other  impacts.  Studies  of  program 
alumni/alumnae  require  ongoing  contact  in  a  highly  mobile  population. 

Case  Studies  explore  a  case  (individual,  group,  or  program)  in  depth.  For 
example,  a  case  study  methodology  might  be  used  to  describe  and  examine  four 
youth  in  a  program  of  fifty  youth  to  carefully  document  program  outcomes. 
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Another  example  might  be  to  use  case  study  methodology  to  examine  two  or 
three  representative  youth  programs  that  grew  out  of  the  YouthALIVE!  program. 

Action  Research  is  conducted  by  educators  to  examine  their  own  practices, 
and  then  to  change  their  practices  based  on  their  findings.  Often  educators  share 
their  findings  and  strategies  to  help  others  in  their  organization  or  field.  Examples 
of  action  research  could  include  an  educator’s  investigation  of  the  use  of  journal 
prompts  by  comparing  a  group  that  gets  prompts  with  one  that  doesn’t,  or  an 
educator’s  investigation  of  the  impact  of  commenting  on  youth  blogs  by 
comparing  youth  engagement  for  a  month  without  comments  and  a  month  with 
comments. 

Data  Collection  Strategies  for  Program  Graduates 

Many  data  collection  strategies  are  standard  for  the  methodologies  described 
above.  Surveys  can  collect  data  from  large  groups  on  paper,  online,  or  by  phone. 
Interviews  in  person  or  by  phone  can  go  into  greater  depth.  Focus  groups  can 
capture  data  from  different  stakeholder  groups.  Observation  protocols  can  be 
developed  for  collecting  data  in  a  variety  of  settings. 

The  largest  challenge  to  data  collection  with  youth  programs,  however,  is 
capturing  data  on  program  graduates.  Youth,  particularly  those  who  are  moving 
into  college  and  adulthood,  are  highly  mobile  and  difficult  to  track.  How  can 
program  graduates  from  a  diverse  group  of  programs  be  reached  for  surveys, 
interviews,  focus  groups,  etc.? 

In  piloting  the  online  alumni  survey  of  the  YES  Program  (Klein,  2013),  many 
hours  (over  100)  of  YES  staff  time  went  into  locating  and  contacting  former  YES 
Teens.  Of  the  288  alumni  and  alumnae  in  the  current  database  (which  is 
incomplete  according  to  YES  staff),  almost  200  are  in  the  Facebook  alumni 
group.  However,  after  posting  notices  on  Facebook  and  sending  individual 
emails  to  about  100  former  YES  Teens,  only  22  completed  the  survey  by  the 
deadline.  This  less  than  one  percent  capture  rate  points  to  two  issues: 
challenges  in  reaching  alumni  and  challenges  in  surveying  alumni. 

To  reach  alumni  requires  tracking  and  data  collection  by  the  program  staff, 
something  not  previously  done  for  the  YES  Program  and  perhaps  many  other 
youth  STEM  OST  education  programs.  Locating  a  large  portion  of  the  alumni 
was  possible  through  Facebook;  however,  other  social  media  would  have  helped. 
Using  alumni  who  remained  in  contact  with  program  staff  to  reach  additional 
alumni  who  in  turn  reached  additional  alumni  was  critical.  To  maintain  contact 
and  encourage  more,  there  needed  to  be  some  reason  for  alumni  to  stay 
connected  with  the  program.  Efforts  to  establish  an  Alumni  Association  helped 
with  YES,  but  it  was  never  clear  how  alumni  would  benefit  from  participation. 

To  collect  data  from  alumni  requires  instruments  that  meet  the  needs  of  the 
former  participants.  The  online  survey  of  YES  alumni  met  the  need  for 
asynchronous,  electronic  communication  -  fill  it  out  any  time  and  anywhere. 
Unfortunately,  there  were  no  incentives  to  complete  the  survey,  and  the  length  of 
the  survey  was  too  long. 


8 


A179 


To  address  these  challenges,  connecting  with  alumni  and  choosing 
appropriate  methods  of  data  collecting,  the  following  strategy  is  recommended  to 
capture  data  from  former  program  participants  of  youth  programs. 

•  Create  a  web-based  presence  for  alumni  that  benefits  alumni  -  much  like 
a  college  alumni  association  with  networking  opportunities,  job  notices, 
stories  from  alumni,  stories  about  the  current  program,  and  opportunities 
to  give  back  to  the  program  (through  suggestions,  resources,  volunteer 
time,  and  even  financial  contributions) 

•  Utilize  a  wide  variety  of  social  media  to  encourage  alumni  to  reach  other 
alumni  -  Facebook,  Twitter,  Linkedln,  Instagram,  etc. 

•  Streamline  an  online  survey  to  capture  minimal  information  as  a  baseline 
-  this  can  always  be  followed  by  another  survey  or  interviews 

•  Offer  incentives  to  complete  the  survey  -  gift  cards  or  chances  to  win  gift 
cards  or  other  items,  t-shirts  or  other  program  related  items,  memberships 
to  the  institution,  etc. 

Grant  Management  Strategies 

Federal  and  foundation  grants  focus  on  research  and  dissemination  rather 
than  program  operation  in  general.  Building  on  the  goal  of  creating  a  sustainable 
YES  Program  model,  it  is  important  to  consider  the  benefits  of  external  funding. 
With  research  and  scale-up  efforts  come  additional  responsibilities  for  staff 
members  that  must  be  added  to  the  existing  core  of  staff,  responsibilities  that 
should  be  covered  by  grant  funding  and  should  end  when  funding  ends.  It  will  be 
important  to  keep  these  separate  from  the  ongoing  daily  costs  of  program 
operation.  Responsibilities  to  address  the  research  questions  above  could 
include  the  following. 

•  Grants  management 

•  Project  management 

•  Project  Coordination 

•  Data  management 

•  Data  entry 

•  Research 

•  Evaluation 

•  Web  design 

•  Web  management 

•  Production  services 

When  stakeholders  ask  for  Return  on  Investment  (ROI)  figures,  this 
recommended  grant  management  strategy  will  become  important.  It  should  be 
noted,  however,  that  not  all  educational  research  supports  the  notion  of  ROI. 
Ongoing  application  of  ROI  can  lead  to  abandoning  program  elements,  based  on 
cost,  that  provide  the  subtle  connections  among  primary  factors  that  make  them 
work.  Much  of  the  research  suggested  in  this  report  examines  complex  systems 
that  cannot  demonstrate  causal  links  and  cannot  be  easily  reduced  to  dollar 
amounts  in  ROI  calculations. 
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ADDITIONAL  FUNDING  SOUGHT 


Two  research  studies  were  proposed  to  the  National  Science  Foundation 
(NSF)  to  study  the  impact  of  the  YES  Program,  the  first  a  study  of  the  impact  of 
the  entire  YES  Program  using  a  comparison  group  and  the  second  a  study  of 
identity  development  in  a  single  group  of  YES  Teens  engaged  in  work  with  DNA 
in  primates. 

YES  Impacts:  A  Study  of  the  Existing  Youth  Exploring  Science  Program 

was  submitted  in  March  2011  to  the  National  Science  Foundation’s  Transforming 
STEM  Learning  (TSL)  program  to  study  the  Youth  Exploring  Science  (YES) 
program  at  the  SLSC.  The  goals  of  YES  Impacts  were  to:  1 )  identify  essential 
programmatic  elements  of  the  YES  Program;  2)  identify  the  impact  of  those 
elements  on  participants;  and,  3)  identify  those  elements  suitable  for  replication 
in  other  settings.  These  goals  were  to  be  accomplished  through  two  phases  of 
research,  a  clinical  interview  study  and  a  short-range  longitudinal  study  using  an 
accelerated  longitudinal  design.  Principal  Investigators  were  Diane  Miller  (SLSC), 
Robert  Tai  (University  of  Virginia)  and  Xitao  Fan  (also  at  UV). 

Research  questions  in  Phase  1  were:  1)  what  do  youth  report  as  the  most 
important  factors  informing  their  decisions  to  participate  in  YES;  and,  2)  what  do 
youth  report  as  the  most  important  factors  in  their  decisions  to  continue  their 
participation  in  YES  for  the  full  four-year  program?  Methods  included  individual 
interviews  with  participants  and  those  waitlisted.  Research  questions  in  Phase  2 
addressing  current  YES  Teens  and  a  comparison  group  of  non-YES  teens  who 
expressed  interest  in  participating  in  YES  were:  1)  over  a  two-year  period,  what 
are  the  educational  trajectories  of  youth  participating  in  YES  and  youth  in  the 
comparison  group;  and,  2)  what  are  the  interests  and  engagements  of  YES  and 
comparison  group  participants  over  a  two-year  period  with  respect  to  STEM 
related  topics  as  related  to  scientific/technical  workforce  YES  Impacts 
development?  Based  on  questionnaire  results,  data  were  to  include  enrollment  in 
non-required  and  advanced  STEM  courses  and  youth’s  performance  in  these 
classes.  This  project  did  not  receive  funding. 

The  Exploring  Identity  project  was  submitted  to  the  National  Science 
Foundation’s  Informal  Science  Education  (ISE)  program  in  December  201 1 .  This 
collaboration  among  the  Sam  Noble  Oklahoma  Museum  of  Natural  History  (lead 
institution),  the  Oklahoma  City  Zoo,  Saint  Louis  Science  Center,  Saint  Louis  Zoo, 
Project  Exploration  and  Oklahoma  City  Community  College  proposed  to  engage 
groups  of  teenagers  in  Oklahoma  City,  St.  Louis  and  Chicago  to  explore  the 
fundamental  question  “What  does  it  mean  to  be  human?”  through  a  series  of 
linked  immersion  projects  in  primatology,  including  both  observations  of  primate 
groups  in  zoos  and  cross-species  DNA  analyses  conducted  in  the  laboratory. 

We  hypothesized  that  participating  teens  would  gain  a  greater  understanding 
of  what  it  means  to  be  human  and,  in  turn,  clarify  and  embrace  their  personal 
identities  through  their  participation.  Exploring  Identity  research,  to  be  conducted 
by  Klein  Consulting,  was  designed  to  explore  the  question:  How  are  encounters 
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with  science  used  in  identity  development  and  transformation  when  youth  explore 
what  it  means  to  be  human?  This  project  was  not  funded.  Project  leaders 
explored  options  for  resubmitting  the  proposal,  but  with  many  project  leaders 
either  retiring  or  leaving  their  institutions,  the  project  did  not  move  forward. 


SUMMARY 

Evaluation  of  the  Community  STEM  Outreach  project,  funded  through  the 
Office  of  Naval  Research  (ONR),  and  a  synthesis  of  previous  YES  Program 
evaluation  reports  suggested  a  number  of  research  questions  focused  on  the 
impact  of  youth  STEM  education  programs  in  out-of-school  time  and  the  strategic 
impact  of  sharing  the  YES  Program  Model.  From  those  questions,  a  number  of 
methodologies  emerged  as  most  relevant.  Rather  than  identify  specific  questions 
and  methodologies  to  pursue,  examples  were  provided  to  assist  the  Saint  Louis 
Science  Center,  Office  of  Naval  Research,  and  others,  and  to  serve  as  a 
foundation  for  developing  future  studies. 

Collecting  data  from  program  graduates  was  identified  as  a  critical  challenge 
to  the  success  of  many  long-term  impact  studies.  Drawing  on  the  experience  with 
the  YES  Program,  the  following  was  recommended  when  graduate  data  were  not 
kept  by  the  program. 

•  Create  a  web-based  presence  for  alumni  that  benefits  alumni 

•  Utilize  a  wide  variety  of  social  media  to  encourage  alumni  to  reach  other 
alumni 

•  Streamline  an  online  survey  to  capture  minimal  information  as  a  baseline 

•  Offer  incentives  to  complete  the  survey 

Grant  management  strategies  were  suggested  to  guide  program  leaders  who 
choose  to  move  forward  with  research  efforts  that  involve  grant  funding.  These 
included  the  addition  of  many  grant  management  responsibilities  to  staff  roles. 
Such  responsibilities  would  also  help  when  potential  funders  ask  for  Return  on 
Investment. 

Two  research  proposals  were  submitted  previously  to  the  National  Science 
Foundation  (NSF)  to  study  YES  Program  impacts.  Neither  was  funded.  Research 
questions  in  one  study  included:  1 )  what  do  youth  report  as  the  most  important 
factors  informing  their  decisions  to  participate  in  YES;  2)  what  do  youth  report  as 
the  most  important  factors  in  their  decisions  to  continue  their  participation  in  YES 
for  the  full  four-year  program?  3)  over  a  two-year  period,  what  are  the 
educational  trajectories  of  youth  participating  in  YES  and  youth  in  the  comparison 
group;  and,  4)  what  are  the  interests  and  engagements  of  YES  and  comparison 
group  participants  over  a  two-year  period  with  respect  to  STEM  related  topics  as 
related  to  scientific/technical  workforce  YES  Impacts  development? 

The  second  proposed  study  hypothesized  that  participating  teens  in  a 
primatology-  focused  program  would  gain  a  greater  understanding  of  what  it 
means  to  be  human  and,  in  turn,  clarify  and  embrace  their  personal  identities 


11 


A182 


through  their  participation.  This  research  addressed  the  question:  How  are 
encounters  with  science  used  in  identity  development  and  transformation  when 
youth  explore  what  it  means  to  be  human? 

At  the  time  of  writing  this  report,  the  authors  were  not  aware  of  plans  for 
additional  proposals  to  NSF  or  ONR  for  related  research. 

This  report  was  developed  to  offer  suggestions  for  future  research.  Questions 
regarding  this  report  can  be  addressed  to  the  authors. 
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informalscience.org/images/research/Sneider_%20Burke_LegacyofYouth 
ALIVE.pdf 
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Community  STEM  Outreach 
Proposed  Dissemination  of  YES 

Submitted  by  Klein  Consulting 
September  201 3 


INTRODUCTION 

In  its  proposal  to  the  Office  of  Naval  Research  (ONR)  dated  August  4,  2010,  the 
Saint  Louis  Science  Center  (SLSC)  proposed  to  “create  a  strategic  plan  for  national 
expansion  of  the  SLSC  Community  STEM  Outreach  Program"  and  to  “identify 
resources,  including  national  partners,  for  national  outreach.”  This  current  document 
outlines  a  strategic  plan  for  the  dissemination  of  the  YES  Program  model  nationally  and 
reports  on  efforts  to-date  to  identify  resources  and  partners. 

In  December  201 1 ,  the  SLSC  contracted  with  Klein  Consulting  to  provide  a  strategic 
plan  for  dissemination.  As  the  external  evaluator  on  the  Community  STEM  Outreach 
project,  Klein  Consulting  was  able  to  draw  on  the  results  of  the  April  2013  meeting  with 
national  partners  and  the  evaluation  to  create  this  strategic  plan.  This  document 
represents  the  work  and  views  of  Klein  Consulting.  Any  opinions,  findings,  and 
conclusions  or  recommendations  expressed  in  this  document  are  those  of  the  authors1 
and  do  not  necessarily  reflect  the  views  of  the  SLSC  or  the  ONR. 


ORIGINAL  NATIONAL  EXPANSION  GOAL 

The  original  intent  of  the  ONR  in  funding  the  Community  STEM  Outreach  project  at 
SLSC  was  to  document  a  mature,  successful  youth  development  program,  the  Youth 
Exploring  Science  (YES)  Program,  and  then,  with  additional  ONR  funding,  disseminate 
the  model  nationally  to  other  science  centers  and  museums.  The  trajectory  of  this  initial 
plan  was  established  to  benefit  SLSC,  ONR,  and  the  field  of  science  learning  in  out-of¬ 
school  time  (OST).  Through  the  Community  STEM  Outreach  Project,  potential  partners 
would  come  together  to  create  a  collaboration  and  plan  for  the  dissemination  of  the 
model,  including  application  for  a  new  ONR  grant. 

Central  to  the  original  plan  was  the  creation  of  the  Institute  for  Science  Learning  at 
SLSC.  The  Institute  was  to  serve  as  the  hub  for  disseminating  the  YES  model,  building 
on  earlier  models  of  the  YouthALIVE!  network  established  through  the  Association  of 
Science-Technology  Centers  (ASTC)  and  internships  created  through  the  St.  Louis 
Center  for  Inquiry  in  Science  Teaching  and  Learning  (CISTL).  Through  workshops, 
symposia,  internships,  and  other  opportunities  at  the  Institute,  practitioners  would  spend 
time  in  St.  Louis  gaining  first-hand  experience  with  the  YES  Program  and  outreach 
activities  to  create  similar  models  in  their  own  local  communities.  The  goal  was  for  the 


1  Questions  regarding  this  document  may  be  addressed  to  the  authors:  Christine  Klein,  Principal,  Klein 
Consulting  (ckleinconsulting@gmail.com),  and  Carey  Tisdal,  Director,  Tisdal  Consulting 
(ctisdal@sbcglobal.net). 
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Institute  to  move  to  a  franchise  model  or  train-the-trainer  model  after  careful  and 
thoughtful  planning. 

In  addition  to  the  Institute,  dissemination  was  planned  through  Science  Beyond  the 
Boundaries  SM  (SBB),  a  network  of  science  centers  founded  by  SLSC  in  2006  “to 
connect  science  center  visitors  with  the  advancing  frontiers  of  science  and  facilitate  the 
connection  between  scientific  research  and  their  lives.”  The  original  goal  was  to  extend 
outreach  “nationally  to  all  areas  of  the  country,  including  those  without  current  Navy 
reach,”  with  SBB  and  the  Institute  as  the  vehicles  for  dissemination. 

The  Changing  Context 

Several  environmental  factors  influenced  the  need  to  adjust  the  initial  trajectory: 
leadership  and  staff  changes,  and  the  national  economic  and  political  landscape. 

Since  submission  of  the  Community  STEM  Outreach  proposal  to  ONR  in  2010,  SLSC 
has  seen  three  different  presidents  and  made  significant  financial  adjustments.  SLSC  is 
now  at  a  new,  and  perhaps  more  stable,  stage  focused  on  sustainability  rather  than 
growth,  including  with  the  YES  Program.  YES  Program  staff  reductions  and  changes, 
along  with  the  resignation  of  YES  founder  and  original  Community  STEM  Outreach 
Principal  Investigator,  Diane  Miller,  occurred  during  this  shift  in  focus.  At  ONR,  the 
Program  Officer  changed  during  this  period.  With  each  change,  institutional  memory 
has  been  lost. 

Local  changes  coincided  with  changes  in  national  politics  and  the  economy. 
Sequestration  has  had  a  profound  affect  on  the  ability  of  science  centers  and  museums 
to  obtain  federal  funding  for  STEM  education.  Federal  plans  to  consolidate  such  funding 
programs  have  caused  science  centers  to  rethink  their  development  strategies.  Grant 
programs  once  focused  on  equity  and  diversifying  the  STEM  workforce  are  shifting. 

As  a  result  of  changes  at  SLSC,  the  Institute  is  no  longer  part  of  the  institution’s 
strategic  plan.  Related  to  changes  at  the  national  level,  SLSC  leaders  decided  not  to 
move  forward  on  another  proposal  to  the  ONR. 

National  Collaboration 

Under  the  leadership  of  Diane  Miller,  SLSC  identified  national  science  center  and 
museum  partners  with  existing  youth  programs  or  plans  for  such  programs. 
Representatives  from  these  nine  national  partners  joined  SLSC  staff,  representatives 
from  the  ONR,  and  local  partners  on  April  21-23,  2013  in  St.  Louis  to  discuss  the  YES 
Program  model  and  its  potential  dissemination.  (See  Appendix  A  for  the  agenda,  and 
Appendix  B  for  a  list  of  participants.) 

On  the  second  day  of  the  meeting,  three  breakout  groups  met  to  begin  discussing 
possible  areas  for  collaboration:  Science  Learning,  Youth  Development,  and  Workforce 
Development.  A  conceptual  model  (Figure  1)  was  discussed,  and  the  focus  of 
discussion  shifted  to  ways  to  bring  the  three  together  -  to  increase  the  “sweet  spot”  at 
the  center. 
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While  there  was  no  consensus,  the  group  agreed  to  carry  on  the  conversation  through 
a  Thinkfinity  group  discussion,  to  be  organized  by  one  volunteer  from  a  national  partner. 
The  next  step  would  be  for  SLSC  leaders  to  develop  action  items  and  solicit  feedback. 
After  the  April  meeting,  the  Thinkfinity  site  was  established  but  few  meeting  participants 
joined  or  commented.  With  lack  of  response  from  conference  attendees,  change  in 
leadership  of  the  YES  Program  at  the  SLSC,  and  information  from  ONR  about  limited 
funding  opportunities,  the  SLSC  administration  concluded  that  an  appropriate  role  for 
the  institution  would  be  to  collaborate  in  any  emerging  initiations  but  not  to  take  a 
leadership  role  in  obtaining  funding  or  working  toward  consensus. 


MULTIMEDIA  TOOL  DESIGN 

To  fulfill  the  Community  STEM  Outreach  goal  to  document  the  YES  Program  for 
possible  future  expansion,  Klein  Consulting,  in  collaboration  with  Tisdal  Consulting, 
undertook  the  design  of  a  multimedia  tool,  currently  titled  Circles  of  Support.  This  web- 
based  tool  design  brings  together  audio,  video,  photos,  and  text  to  share  the  story  of  the 
YES  Program.  Organized  around  the  YES  Program’s  logic  model  and  growing  out  of  the 
evaluation  of  the  YES  Program,  the  tool  design  allows  users  to  look  at  the  broad  ideas 
that  serve  as  the  program’s  foundation  and  to  zero  in  on  specific  program  aspects  like 
journaling. 

The  Circles  of  Support  design  and  the  accompanying  written  documentation  of  the 
YES  Program  serve  as  the  basis  for  this  current  dissemination  plan. 


CREATING  A  DISSEMINATION  PLAN 

The  dissemination  plan  proposed  here  responds  to  the  changed  environment  and 
builds  on  the  legacy  of  YouthALIVE!  It  is  designed  to  spark  discussion,  and  eventually 
lead  to  consensus  for  future  collaborative  efforts. 

A  study  by  Cary  Sneider  and  Meg  Burke  (201 1 )  provides  an  overview  of  the 
numerous  extended  programs  for  youth  offered  by  science  centers  and  museums  a 
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decade  after  funding  for  YouthALIVE!  ended,  and  characterizes  the  content  and 
structure  of  the  programs.  Sneider  and  Burke  offer  three  recommendations  based  on 
their  findings:  focused  support  for  youth  programs,  support  for  youth  program  leaders, 
and  research.  In  their  first  recommendation,  authors  call  for  leaders  in  the  field  to: 

Support  current  youth  programs  at  museums  and  science  centers,  and  encourage 
the  establishment  and  growth  of  new  programs  to  expand  the  opportunities  for  our 
nation’s  youth  to  experience  the  engagement,  capacity,  and  continuity  they  need  to 
succeed  in  technical  careers.  (Sneider  &  Burke,  201 1 ) 

The  Circles  of  Support  multimedia  tool  developed  through  the  Community  STEM 
Outreach  project  was  designed  to  meet  two  needs:  to  strengthen  the  existing  YES 
Program  at  SLSC  by  creating  a  record  of  the  program’s  underlying  rationale  for  new 
staff,  and  to  provide  the  creation  of  a  tool  to  support  expanded  opportunities  for  youth 
nationally  based  on  the  YES  Program  model.  The  tool  presents  elements  of  the  YES 
Program  as  a  working  model  to  allow  other  institutions  to  select  and  adapt  the  ideas, 
research,  and  best  practices  that  fit  the  context  of  their  local  area.  Target  audiences  for 
the  tool  are  community-based  organizations  and  science  centers  interested  in  starting  a 
new  program  or  improving  an  existing  program.  Units  about  each  program  element 
include  a  video  introduction,  documentation  of  the  YES  Program  element,  sample 
program  documents,  and  references.  An  alpha  version  of  the  tool,  available  through  a 
website,  was  available  for  testing  at  the  conclusion  of  the  grant  period  along  with  a  print 
documentation  available  in  PDF  on  the  website  or  by  request. 


THE  PLAN 

This  dissemination  plan  involves  three  phases:  1)  testing  and  revision  of  the  Circles  of 
Support  multimedia  tool  design;  2)  broad  dissemination  of  the  URL  for  the  website;  and, 
3)  professional  development  of  staff  using  the  multimedia  tool  and  related  resources  as 
a  foundation. 

Phase  1:  Testing  and  Revision 

Within  the  current  ONR  grant,  funds  were  allocated  to  document  the  YES  Program 
model  and  design  the  Circles  of  Support  multimedia  tool.  Any  professional  development 
effort,  like  this  one,  needs  user  testing,  feedback,  and  revision  prior  to  dissemination. 
Thus,  testing  and  revision  is  a  key  first  phase  in  the  dissemination  plan.  This  Phase  1 
involves  three  steps. 

Usability  Testing.  With  an  audience  of  staff  from  programs  and  institutions  that  might 
use  the  Circles  of  Support  multimedia  tool,  a  card  sort  technique  like  an  Optimal  Sort2, 
would  allow  SLSC  to  know  if  users  can  find  information  on  the  site.  Results  from  this 
along  with  feedback  collected  from  the  website’s  comment  form,  should  then  be  used  to 
revise  the  site.  Feedback  from  the  usability  testing  could  also  be  used  to  identify  where 
additional  content  is  needed,  multimedia  elements  need  replacing  or  expanding,  and 


2  http://www.optimalworkshop.com/ 
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where  users  would  benefit  from  more  or  less  information. 


Case  Studies.  To  go  further  in  determining  usability  of  the  Circles  of  Support 
resource,  three  case  studies  with  staff  from  three  different  programs  would  provide 
insight.  These  three  programs  should  include  one  existing  youth  program,  one  new 
program  underdevelopment,  and  the  YES  Program  (in  new  staff  orientation).  Such  case 
studies  would  identify  uses  and  usefulness,  in  addition  to  suggesting  further  site 
changes. 

IT  Support  Assessment.  SLSC  needs  to  determine  whether  the  structure  and  design 
of  the  Circles  of  Support  multimedia  tool  can  be  updated  and  revised  by  existing  YES 
staff  and  resources,  or  whether  the  site  needs  to  be  turned  over  to  a  website 
development  professional  (internal  or  external)  to  rebuild  and  revise.3 

Phase  2:  Broad  Dissemination  of  Circles  of  Support 

Once  testing  and  revision  have  yielded  a  stable  and  effective  website,  broad 
dissemination  of  the  URL  is  possible.  In  addition  to  the  Science  Beyond  the 
Boundariessm  network,  the  link  with  an  accompanying  letter  should  be  distributed 
through  the  ISEN  listserv,  ASTC’s  Youth  Program  Network,  and  the  ONR  Stem2Stern 
program.  A  link  on  the  Circles  of  Support  home  page  to  solicit  feedback,  should  be 
monitored  to  ensure  the  stability  and  effectiveness  of  the  site  throughout  this  phase. 

Phase  3:  Professional  Development  for  National  Audiences 

The  Circles  of  Support  multimedia  tool  provides  a  foundation  for  the  development  of 
new  programs,  strengthening  of  existing  programs,  and  training  of  staff.  By  itself, 
however,  the  website  doesn’t  go  far  enough  on  training.  The  videos  are  designed  to 
spark  interest  and  open  doors  to  thought  and  conversation.  The  original  idea  for  the 
Institute  for  Science  Learning  established  a  national  training  and  research  program  in 
St.  Louis  to  carry  on  that  conversation.  While  such  an  ambitious  plan  as  the  Institute  is 
not  necessary,  it  would  be  wise  to  build  on  the  strengths  of  the  YES  staff  and  offer 
training  to  others.  Workshops  and  internships  would  take  the  Circles  of  Support 
multimedia  tool  to  the  next  level. 

Such  a  fee-for-service  opportunity  for  national  partners’  staff  would  strengthen 
existing  YES  staff  and  could  provide  funding  for  the  maintenance  of  the  Circles  of 
Support  multimedia  tool.  A  business  plan  and  curriculum  would  need  development.  We 
recommend  that  the  curriculum  be  organized  around  the  development  of  logic  models4, 
the  central  organizing  strategy  used  for  the  architecture  of  the  Circles  of  Support 
multimedia  tool.  Logic  models,  because  of  their  increasingly  wide-spread  use  (e.g.  The 


3  The  biggest  limitation  of  the  Alpha  version  of  the  site  is  the  lack  of  a  search  mechanism.  It  is 
possible  for  this  to  be  added  to  the  site  with  some  additional  internal  or  external  web 
development  expertise  and  consultation.  The  advantage  of  the  current  system,  i.e.  the  ease  of 
revision  and  update  by  a  YES  staff  member,  would  need  to  be  balanced  against  moving  to  a 
more  sophisticated  platform. 

4  Currently,  this  expertise  does  not  exist  among  the  group  of  YES  staff  members.  It  could  be 
obtained  through  collaboration  with  other  SLSC  departments,  consultants,  or  investment  of  staff 
time  in  online  training. 
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Community  Tool  Box,  2013),  particularly  in  the  development  of  funding  and  evaluation 
efforts,  should  be  an  attractive  framework  for  program  staff,  development  staff,  and 
administrators  at  community-based  organizations  and  science  museums.  Evaluation 
should  be  an  integral  piece. 


RESOURCES  NEEDED 

Resources  (time,  personnel,  supplies,  etc.)  will  be  required  to  carry  out  each  phase  of 
this  proposed  dissemination  plan.  These  include  the  following: 

•  Circles  of  Support  Website  Manager 

•  Circles  of  Support  Hosting  and  Maintenance 

•  Field  Testing  Staff  or  Consultants 

•  Dissemination  Leader 

•  Professional  Development  Developers  and  Facilitators 

For  the  web-based  multimedia  tool  to  continue  and  grow  as  a  resource  for  SLSC  staff 
and  national  STEM  OST  educators,  someone  will  need  to  maintain  and  continue  to 
update  the  Circles  of  Support  website.  The  site  was  designed  with  ease  of  maintenance 
and  revision  in  mind  using  a  web-authoring  program  that  does  not  require  knowledge  of 
HTML  and  allows  website  revisions  and  updates  based  on  a  modest  investment  of  time 
in  learning  the  software.  The  website  manager  will  need  minimal  understanding  of 
website  development,  and  perhaps  more  importantly,  this  person  will  need  a  strong 
understanding  of  1)  the  YES  Program  model  conveyed  through  the  site,  2)  science 
center  and  OST  programs  nationally,  and  3)  the  staff  that  support  them.  The  website 
manager  must  serve  as  the  interface  between  the  program  model,  the  users,  and  the 
technical  side  of  the  site.  HTML  programming  for  the  site  can  be  done  in-house  or 
through  consultants,  but  the  manager  needs  to  know  the  users  and  the  rationale  behind 
the  website  to  keep  it  viable. 

As  with  any  website,  hosting  and  maintenance  are  ongoing  expenses  which  should  be 
included  in  any  plan.  These  costs  can  be  kept  to  a  minimum. 

The  field  testing,  case  studies,  and  IT  assessment  described  in  Phase  1  of  the  plan 
require  expertise.  This  can  be  carried  out  in-house  if  qualified  staff  exist  and  have  the 
time,  or  it  can  be  completed  by  consultants.  Field  testing  in  the  initial  stages  will  greatly 
improve  the  usefulness  and  effectiveness  of  the  website  and  dissemination  of  the 
model. 

For  the  dissemination  to  move  forward,  a  staff  member  needs  the  time  and  additional 
resources  to  focus  on  dissemination.  A  staff  member  responsible  for  leadership  of  this 
effort  will  greatly  increase  the  effectiveness  of  the  dissemination,  as  will  the 
administrators  to  hold  this  person  accountable.  This  person  will  need  to  serve  as  point- 
person  for  SBB,  ISEN  members,  ASTC,  and  ONR  throughout  Phase  2. 

If  SLSC  moves  forward  with  Phase  3,  YES  staff  and  experts  (internal  to  SLSC  or 
external)  will  be  needed  to  develop  the  business  model  and  curriculum,  recruit 
participants,  support  the  program’s  logistics,  and  facilitate  the  learning  experiences. 
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SUMMARY 


Dissemination  of  the  YES  Program  model  will  provide  other  organizations  with  a  much 
needed  resource.  By  testing,  revising,  and  disseminating  the  Circles  of  Support 
multimedia  tool,  SLSC  has  the  potential  to  make  a  meaningful  and  long-term  impact  on 
the  field  of  informal  STEM  education.  By  building  on  the  information  shared  on  the 
website  and  in  the  accompanying  documentation  with  workshops  and  internships  for 
OST  staff  from  across  the  nation,  SLSC  has  the  opportunity  to  share  its  story  in  an  even 
deeper  and  more  impactful  way. 
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APPENDIX  A  -  Agenda  for  April  2013  Meeting 


Sunday,  April  21  (SLSC-  James  S.  McDonnel  Planetarium,  enter  from  Forest  Park} 

5:00  pm  Cocktails  and  Hors  d'oeuvres 
6:00  pm  Dinner 

7:00  pm  Welcome  by  Bert  Vescolani 

Introduction  of  keynote  speaker  by  Diane  Miller 
Keynote  Speaker:  Alan  Friedman 

Monday,  April  22  (4900  Manchester,  Taylor  Community  Science  Resource  Center} 

8:30  am  Breakfast 

9am-12  pm  Welcome  by  Bert  Vescolani,  President  (Saint  Louis  Science  Center) 

Overview  of  Symposium,  Diane  Miller 
Introductions 
Tour  of  the  facility 
Welcome  by  Lee  Metcalf 
Welcome  from  the  Office  of  Naval  Research 
12-1  pm  Lunch  at  Taylor 

1-5  pm  Community  STEM  Outreach  Presentation  by  Klein  Consulting 
Break 

Museum  Presentations:  How  Do  Museums  engage,  inspire  and 

inform  youth  in  STEM  education?  How  do  they  engage  the  community? 

Break 

What  are  some  of  the  topics  we  need  to  address  moving  forward  as  a 
collaborative? 

Announcements 

5:30  pm  Dinner  at  Joyia  (4501  Manchester  Avenue,  314.531.5300) 

Shuttle  service  will  be  available 

Tuesday,  April  23  (Taylor  Community  Science  Resource  Center) 

8  am  Breakfast  at  Taylor  Community  Science  Center 

9  am  Continuing  the  Conversation  on  community  STEM  youth  programs 

Expanding  the  Alliance  -  Working  together  to  increase  STEM  in  the  Community. 
Small  Group  Session  Topics  include:  Rationale  for  collaboration,  Categories  of 
collaboration,  Barriers  to  collaboration 
Large  Group  Discussion 
Noon  Lunch  at  Taylor 

lpm  Framing  an  Action  Plan,  Steps  for  the  Future 

3pm  Symposium  Concludes 
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APPENDIX  B  -  Participants  at  April  2013  Meeting 


Keynote  Speaker: 

•  Museum  Development  and  Science  Communication  -  Alan  J.  Friedman 
Office  of  Naval  Research: 

•  CDR  Joseph  Cohn 

•  Anastazia  Aguilar 

National  Partners: 

•  Bishop  Museum  -  Blair  Collins 

•  California  Science  Center  -  Jeff  Rudolph 

•  Great  Lakes  Science  Center  -  Whitney  Owens 

•  Lawrence  Hall  of  Science  -  Elizabeth  Stage  and  Erica  Barrueto 

•  Museum  of  Life  and  Science  -  Troy  Livingston 

•  National  Museum  of  American  History,  Smithsonian  Institution  -  Judy  Gradwhol 
and  Carrie  Kotcho 

•  New  Mexico  Museum  of  Natural  History  and  Science  -  Charles  Walter  and  Deb 
Novak 

•  Reuben  Fleet  Science  Center  -  Steven  Snyder  (formerly  of  The  Franklin 
Institute) 

•  Science  Museum  of  Minnesota  -  Anika  Ward  and  Joanne  Jones-Rizzi 
St.  Louis  Partners: 

•  Saint  Louis  Science  Center  -  Bert  Vescolani,  Pat  Williams,  Diane  Miller,  Siinya 
Williams 

•  Missouri  Botanical  Gardens  -  Shelia  Voss 

•  Saint  Louis  Zoo  -  Louise  Bradshaw 

•  Daugherty  Group  and  US  Navy  -  RADM  Lee  Metcalf 

•  Hosco  Farms  -  Gibran  Burchett 

•  Klein  Consulting  -  Christine  (Kit)  Klein 

•  Tisdal  Consulting  -  Carey  Tisdal 
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